![A silhouette of a dump truck dumping trash bags.](https://raw.githubusercontent.com/matt-dray/stickers/master/output/ghdump_hex.png)
My garbage GitHub repos being dumped onto my local machine (via openclipart.org, CC0 1.0)
tl;dr
Run ghd_copy()
from the {ghdump} package to either clone or download all the GitHub repositories for a given user. Intended for archival purposes or setting up a new computer.
The package comes with no guarantees and will likely be in a perpetual work-in-progress state. Please submit issues or pull requests.
Clone army
Situation:
- Sometimes I get a new computer and want to clone all my repos to it
- Sometimes I want to be able to archive my repos so I’m not dependent on GitHub nor any given computer
- it would be tedious to download or clone the repos one-by-one from the GitHub interface
Wants:
- To clone (with HTTPS or SSH) or download all of my repos with one command
- Be able to unzip downloaded repos en masse if I want to
- Do all this from within R, mostly for the learning experience, but also to allow for user interactivity
Observations:
- I don’t know of a specific R function that automates mass-downloading or mass-cloning of GitHub repos
- the {gh} package provides a lightweight GitHub API wrapper for R that’s likely to be helpful
- R has many file-handling functions that will be helpful
{ghdump}
The result is that I wrote a function, ghd_copy()
, that copies (clones or downloads) all the repos for a given user to a specified location. You can get it in the tiny {ghdump} package.
The function interacts with the GitHub API thanks to the {gh} package by Gábor Csárdi, Jenny Bryan and Hadley Wickham, while iterating over repos comes thanks to the {purrr} package by Lionel Henry and Hadley Wickham.
Update
As of May 2022 there’s also a handy rOpenSci package called {gitcellar}, by Maëlle Salmon and Jeroen Ooms, which is for downloading an organisation’s repos for archival purposes.
Get and use
Install with:
remotes::install_github("matt-dray/ghdump")
To use the package, you’ll need a GitHub account and a GitHub Personal Access Token (PAT) stored in your .Renviron file. You can do this with the following steps:
usethis::browse_github_pat() # opens browser to generate token
usethis::edit_r_environ() # add your token to the .Renviron
# then restart R
You can use {ghdump} to download the repos for a specified user:
ghdump::ghd_copy(
gh_user = "matt-dray", # download repos for this user
dest_dir = "~/Documents/repos", # full local file path to copy to
copy_type = "download" # "download" or "clone" the repos
)
Or clone them:
ghdump::ghd_copy(
gh_user = "matt-dray",
dest_dir = "~/Documents/repos"
copy_type = "clone",
protocol = "https" # specify "https" or "ssh"
)
If you want to use the SSH protocol when cloning, you need to make sure that you’ve set up your keys.
Interactivity
My expectation is to use ghd_copy()
infrequently and in a non-programmatic way, so I’ve made it quite interactive. This means user input is required; you’ll get some yes/no questions in the console that will affect how the function runs.
Here’s an imaginary demo of the output from ghd_copy()
when copy_type = "download"
:
> ghd_copy("made-up-user", "~/Desktop/test-download", "download")
Fetching GitHub repos for user made-up-user... 3 repos found
Create new directory at path ~/Desktop/test-download? y/n: y
Definitely download all 3 repos? y/n: y
Downloading zipped repositories to ~/Desktop/test-download
trying URL 'https://github.com/made-up-user/fake-repo-1/archive/master.zip'
Content type 'application/zip' length 100 bytes
==================================================
downloaded 100 bytes
trying URL 'https://github.com/made-up-user/fake-repo-2/archive/master.zip'
Content type 'application/zip' length 100 bytes
==================================================
downloaded 100 bytes
trying URL 'https://github.com/made-up-user/fake-repo-3/archive/master.zip'
Content type 'application/zip' length 100 bytes
==================================================
downloaded 100 bytes
Unzip all folders? y/n: y
Unzipping repositories
Retain the zip files? y/n: y
Keeping zipped folders.
Remove '-master' suffix from unzipped directory names? y/n: y
Renaming files to remove '-master' suffix
Finished downloading
And now imaginary demo of the output from ghd_copy()
when copy_type = "clone"
:
> ghd_copy("made-up-user", "~/Desktop/test-clone", "clone", "ssh")
Fetching GitHub repos for user made-up-user... 3 repos found
Create new directory at path ~/Desktop/test-clone? y/n: y
Definitely clone all 3 repos? y/n: y
Cloning repositories to ~/Desktop/test-clone
Cloning into 'fake-repo-1'...
Cloning into 'fake-repo-2'...
Cloning into 'fake-repo-3'...
Finished cloning
Note that cloning has only been tested on my own Mac OS machine at this point (June 2020) and is not guaranteed to work elsewhere yet. Please submit issues or pull requests to help improve this.
Under the hood
What are the steps to downloading repos with ghdump::ghd_copy()
?
First, to get repo info:
ghdump:::ghd_get_repos()
passes a GitHub username togh::gh()
, which contacts the GitHub API to return a gh_response object that contains info about each of that user’s reposghdump:::ghd_extract_names()
takes the gh_response object fromghd_get_repos()
and extracts the names into a character vector
Then to download (if copy_type = "download"
):
ghdump:::ghd_enframe_urls()
turns the character vector of repo names into a data.frame, with a corresponding column that contains the URL to a zip file for that repoghdump:::ghd_copy_zips()
takes each zip file URL from that data frame and downloads them to the file path provided by the userghdump:::ghd_unzip()
unzips the zipped repos
You can, of course, use these intermediate functions if you have slightly different needs. Maybe you want to limit the repos that are downloaded; do this by filtering the vector output from ghdump:::ghd_extract_names()
for example.
Or to clone (if copy_type = "clone"
):
ghdump:::ghd_clone_multi()
that iterates cloning over the repos, itself callingghdump:::ghd_clone_one()
Why bother?
What did I learn from doing this?
Iteration
Aside from {gh}, the package also depends on {purrr} for iterative programming.
For example, the gh_response object output from ghdump:::ghd_get_repos()
is passed to map()
with the pluck()
function to extract the repo names.
Another example is the use of walk()
, which is like map()
, except we use it when the output is some ‘side effect’. By ‘side effect’, we mean that it doesn’t return an R object. For example, we can walk()
the unzip()
function over the path to each zip file. This doesn’t return anything in R; it results in some local files being manipulated.
File manipulation
R can be used to interact with files on your computer. There’s a number of these base R functions in the package:
dir.create()
to create a new folderfile.remove()
to remove a file or folderlist.files()
andlist.dirs()
to return a character vector files and folders at some pathfile.rename
to change the name of a file or folderunzip()
to unpack a zipped folder
User input
How do you ask questions of your user and get answers? This interactivity is made possible by readline()
. You pass it a string to prompt the user, whose return value can be stored.
For example, this is how it looks in the console:
> answer <- readline("Do you like pizza? ")
Do you like pizza? yes
> answer
[1] "yes"
Where a user has written yes
after the prompt on the second line.
Stickers
I’ve designed a few hex stickers with the {hexSticker} package; you can see them in my ‘stickers’ GitHub repo. This time I made the sticker for {ghdump} using Dmytro Perepolkin’s {bunny} package, which is a helper for the {magick} package from Jeroen Ooms. It’s a very smooth process with much flexibility.
This belongs in a dump
Yeah, maybe. It’s not sophisticated, but I’ve found it useful for my own specific purposes.
Session info
## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.2.0 (2022-04-22)
## os macOS Big Sur/Monterey 10.16
## system x86_64, darwin17.0
## ui X11
## language (EN)
## collate en_GB.UTF-8
## ctype en_GB.UTF-8
## tz Europe/London
## date 2022-05-15
## pandoc 2.17.1.1 @ /Applications/RStudio.app/Contents/MacOS/quarto/bin/ (via rmarkdown)
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date (UTC) lib source
## blogdown 1.9 2022-03-28 [1] CRAN (R 4.2.0)
## bookdown 0.26 2022-04-15 [1] CRAN (R 4.2.0)
## bslib 0.3.1 2021-10-06 [1] CRAN (R 4.2.0)
## cli 3.2.0 2022-02-14 [1] CRAN (R 4.2.0)
## digest 0.6.29 2021-12-01 [1] CRAN (R 4.2.0)
## evaluate 0.15 2022-02-18 [1] CRAN (R 4.2.0)
## fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.2.0)
## fontawesome 0.2.2 2021-07-02 [1] CRAN (R 4.2.0)
## htmltools 0.5.2 2021-08-25 [1] CRAN (R 4.2.0)
## jquerylib 0.1.4 2021-04-26 [1] CRAN (R 4.2.0)
## jsonlite 1.8.0 2022-02-22 [1] CRAN (R 4.2.0)
## knitr 1.39 2022-04-26 [1] CRAN (R 4.2.0)
## magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.2.0)
## R6 2.5.1 2021-08-19 [1] CRAN (R 4.2.0)
## rlang 1.0.2 2022-03-04 [1] CRAN (R 4.2.0)
## rmarkdown 2.14 2022-04-25 [1] CRAN (R 4.2.0)
## rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.2.0)
## sass 0.4.1 2022-03-23 [1] CRAN (R 4.2.0)
## sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.2.0)
## stringi 1.7.6 2021-11-29 [1] CRAN (R 4.2.0)
## stringr 1.4.0 2019-02-10 [1] CRAN (R 4.2.0)
## xfun 0.30 2022-03-02 [1] CRAN (R 4.2.0)
## yaml 2.3.5 2022-02-21 [1] CRAN (R 4.2.0)
##
## [1] /Library/Frameworks/R.framework/Versions/4.2/Resources/library
##
## ──────────────────────────────────────────────────────────────────────────────