--- title: "Troubleshoot The Hubs" author: "Lori Shepherd" date: "Modified: Sept 2019. Compiled: `r format(Sys.Date(), '%d %b %Y')`" output: BiocStyle::html_document: toc: true vignette: > % \VignetteIndexEntry{Troubleshooting The Hubs} % \VignetteEngine{knitr::rmarkdown} % \VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} knitr::opts_chunk$set(eval = FALSE) ``` # Overview In spring 2019 the Hubs ([AnnotationHub][]/[ExperimentHub][]) upgraded their backend to utilize [BiocFileCache][]. This upgrade changed how resources were downloaded and saved. While the Hub code itself ensures validity, it is possible to access the `BiocFileCache` of resources directly without the Hub front ends, which opens up the possibility of having caching problems. This document will touch on some troubleshooting for these issues as well as any other frequently asked issues. If the question or answer cannot be found here please ask on the [Bioconductor Support Site][] or on the mailing list `` # Troubleshooting ## Accessing Behind A Proxy The ExperimentHub and AnnotationHub use CRAN package `httr` functions `HEAD` and `GET` for accessing web resources. This can be problematic if operating behind a proxy. The easiest solution is to set the `httr::set_config` with the proxy information. ``` proxy <- httr::use_proxy("http://my_user:my_password@myproxy:8080") ## or proxy <- httr::use_proxy(Sys.getenv('http_proxy')) httr::set_config(proxy) AnnotationHub::setAnnotationHubOption("PROXY", proxy) ## or ExperimentHub::setExperimentHubOption("PROXY", proxy) ``` ## Invalid Cache An invalid cache ERROR results from a missing sqlite or index file in the Hub's BiocFileCache. The Hub code needs these files in order to operate correctly. Rerun the Hub constructor (`AnnotationHub()` or `ExperimentHub()`) again. If you were trying to run the constructor with `localHub=TRUE`, you will have to run `localHub=FALSE` at least once to redownload the Hub sqlite file. ## Corrupt Cache A corrupt cache ERROR results from multiple entries in the BiocFileCache matching a query for a particular file. This will involve removing one, multiple, or all entries for a file. Please see specific section below although all follow the same principles. ### sqlite file If the sqlite file is the problematic file you should see something like the following (maybe be experimenthub.sqlite3 respectively) ``` > ah = AnnotationHub() Error: Corrupt Cache: sqlite file See AnnotationHub's TroubleshootingTheHubs vignette section on corrupt cache cache: /home/lori/.cache/AnnotationHub filename: annotationhub.sqlite3 ``` You will need to investigate the underlying BiocFileCache for the Hub and remove some or all of the files so that there is only a single entry for the filename. Call the BiocFileCache constructor with the path listed as `cache` in the ERROR message. ``` library(BiocFileCache) bfc <- BiocFileCache("/home/lori/.cache/AnnotationHub") ``` Now we can query the BiocFileCache using the `filename` of the ERROR message. This shows the number of entries for the filename. There should only be one row ``` res <- bfcquery(bfc, "annotationhub.sqlite3", field="rname", exact=TRUE) res # A tibble: 2 x 10 rid rname create_time access_time rpath rtype fpath last_modified_t… etag 1 BFC1 anno… 2019-03-05… 2019-03-06… /hom… web http… 2019-02-19 19:1… NA 2 BFC13 anno… 2019-03-06… 2019-03-06… /hom… rela… /hom… NA NA # … with 1 more variable: expires ``` You will need to deterime if you can validate which entry should remain by evaluating the entries in the cache; [dplyr package][] methods may be useful for parsing the tibble res object. If you can identify which entry should be kept - remove the other entries in the cache or rename the rname by calling the BiocFileCache functions bfcremove or bfcupdate with the rid of the offending entries ``` library(dplyr) res %>% select(rid, rname, rpath, fpath) # example of rename the rname bfcupdate(bfc, rids="BFC13", rname="What I am renaming the rname to") # example remove bfcremove(bfc, rids="BFC13") ``` If you cannot figure out which entry should be kept I recommend removing all entries so that a fresh redownload can occur. Then call the Hub constructor. ``` bfcremove(bfc, rids=res$rid) ah = AnnotationHub() ``` #### redownload of sqlite file A force redownload of the sqlite hub file can be achieved through the refreshHub function. To specify which of the Bioconductor Hubs to redownload use the hubClass argument with either AnnotationHub or ExperimentHub. ``` ah2 = refreshHub(hubClass="AnnotationHub") ``` ### index file If the index file is the problematic file you should see something like the following (maybe be experimenthub.index.rds respectively) ``` > ah = AnnotationHub() snapshotDate(): 2019-02-19 Error: Corrupt Cache: index file See AnnotationHub's TroubleshootingTheHubs vignette section on corrupt cache cache: /home/lori/.cache/AnnotationHub filename: annotationhub.index.rds ``` You will need to investigate the underlying BiocFileCache for the Hub and remove some or all of the files so that there is only a single entry for the filename. Call the BiocFileCache constructor with the path listed as `cache` in the ERROR message. ``` library(BiocFileCache) bfc <- BiocFileCache("/home/lori/.cache/AnnotationHub") ``` Now we can query the BiocFileCache using the `filename` of the ERROR message. This shows the number of entries for the filename. There should only be one row ``` res <- bfcquery(bfc, "annotationhub.index.rds", field="rname", exact=TRUE) res # A tibble: 2 x 10 rid rname create_time access_time rpath rtype fpath last_modified_t… etag 1 BFC2 anno… 2019-03-05… 2019-03-05… /hom… rela… 66d4… NA NA 2 BFC14 anno… 2019-03-06… 2019-03-06… /hom… rela… /hom… NA NA # … with 1 more variable: expires ``` You will need to deterime if you can validate which entry should remain by evaluating the entries in the cache; [dplyr package][] methods may be useful for parsing the tibble res object. If you can identify which entry should be kept - remove the other entries in the cache or rename the rname by calling the BiocFileCache functions bfcremove or bfcupdate with the rid of the offending entries ``` library(dplyr) res %>% select(rid, rname, rpath, fpath) # example of rename the rname bfcupdate(bfc, rids="BFC14", rname="What I am renaming the rname to") # example remove bfcremove(bfc, rids="BFC14") ``` If you cannot figure out which entry should be kept I recommend removing all entries so that a fresh redownload can occur. Then call the Hub constructor. ``` bfcremove(bfc, rids=res$rid) ah = AnnotationHub() ``` ### resource path If the resource path is a problem, this indicates that there may be duplicate files in the cache. There may only exist at any given time one downloaded file path per resource as indicated by the filename schema "_`. This ERROR indicates duplicate values `_` The ERROR should look similar to ``` Error: Corrupt Cache: resource path See AnnotationHub's TroubleshootingTheHubs vignette section on corrupt cache cache: /home/lori/.cache/AnnotationHub potential duplicate files: 499d6769cf1d_5012 66d42a51a258_5012 ``` You will need to investigate the underlying BiocFileCache for the Hub and remove some or all of the files so that there is only a single entry for the resource path. Call the BiocFileCache constructor with the path listed as `cache` in the ERROR message. ``` library(BiocFileCache) bfc <- BiocFileCache("/home/lori/.cache/AnnotationHub") ``` Now we can query the BiocFileCache using the `duplicate files` of the ERROR message. ``` res <- bfcquery(bfc, "5012", field="rpath", exact=FALSE) res # A tibble: 2 x 10 rid rname create_time access_time rpath rtype fpath last_modified_t… etag 1 BFC3 AH50… 2019-03-05… 2019-03-05… /hom… web http… 2014-03-28 09:2… dd0c… 2 BFC19 dup … 2019-03-06… 2019-03-06… /hom… web http… 2014-03-28 09:2… dd0c… # … with 1 more variable: expires ``` You will need to deterime if you can validate which entry should remain by evaluating the entries in the cache; [dplyr package][] methods may be useful for parsing the tibble res object. If you can identify which entry should be kept - remove the other entries in the cache or rename the rname by calling the BiocFileCache functions bfcremove. If you cannot the best appropate is to remove the resource and redownload a new entry. ``` # remove single entry bfcremove(bfc, rids="BFC19") # remove all bfcremove(bfc, rids=res$rid) ``` If the query resulted in only one entry, there is likely a file in your cache location that has the format similar to the entry in the cache and will have to be removed or renamed. An example ``` Error: Corrupt Cache: resource path See AnnotationHub's TroubleshootingTheHubs vignette section on corrupt cache cache: /home/lori/.cache/AnnotationHub potential duplicate files: 45b42ba7aaa1_38317 7a4726896632_38317 ``` But when you do the query there is only one value ``` > res <- bfcquery(bfc, "38317", field="rpath", exact=FALSE) > res # A tibble: 1 x 10 rid rname create_time access_time rpath rtype fpath last_modified_t… etag 1 BFC37 AH32… 2019-03-08… 2019-03-08… /hom… web http… 2013-07-25 07:0… 11c3… # … with 1 more variable: expires ``` The file path of the valid entry is ``` > bfcinfo(bfc, rid="BFC37") %>% dplyr::select(rpath) # A tibble: 1 x 1 rpath 1 /home/lori/.cache/AnnotationHub/45b42ba7aaa1_38317 ``` Therefore we would want to move or rename the file `7a4726896632_38317`. ``` > fl <- file.path(bfccache(bfc), "7a4726896632_38317") > fl [1] "/home/lori/.cache/AnnotationHub/7a4726896632_38317" > unlink(fl) ``` ### resource id If the resource id is problematic, it generally means that there are entries in the BiocFileCache with the same `rname`. You would see an ERROR similar to the following: ``` > cache(ah[1:4]) adding rname 'AH5015 : 5015' Error: Corrupt Cache: resource id See AnnotationHub's TroubleshootingTheHubs vignette section on corrupt cache cache: /home/lori/.cache/AnnotationHub reason: not all 'rnames' found or unique. ``` You will need to investigate the underlying BiocFileCache for the Hub and remove or rename resources so there are no duplicate rnames in the cache. ``` library(BiocFileCache) bfc <- BiocFileCache("/home/lori/.cache/AnnotationHub") bfcinfo(bfc) %>% dplyr::select(rname) ``` You can remove with `bfcremove` and can rename with `bfcupdate`. ## Corrupt Database A corrupt hub database ERROR results from an invalid hub sqlite file. Perhaps the original download was interrupted or the file was overwritten. You will have to remove the currently downloaded sqlite file so it can be redownloaded. The output you received looked something similar to the following: ``` > ah = AnnotationHub() Error: Corrupt Hub Database; See AnnotationHub's TroubleshootingTheHubs vignette section on corrupt database database: '/home/lori/.cache/AnnotationHub/66d467fcefa5_annotationhub.sqlite3' reason: missing tables ``` The simplest solution is to remove the file listed as the `database:` in the above output. ``` # from the example above file.remove('/home/lori/.cache/AnnotationHub/66d467fcefa5_annotationhub.sqlite3') ``` Rerunning the constructor should now redownload a valid database object. ## Cannot retrieve resource This occurs when using a Hub object designated to only access locally downloaded files (ex. hub <- `AnnotationHub(localHub=TRUE)`) and a resource cannot be found in the current BiocFileCache database. The ERROR looks similar to ``` Error: Cannot retrieve resource Rerun constructor with 'localHub=FALSE' or exclude ID Requested resource not found in local cache: AH66165 : 72911 ``` The options are to change the Hub object to `isLocalHub=FALSE` (`isLocalHub(hub)<-FALSE`) so the file can be downloaded. If this cannot be done because of internet access or other issues, the resource will not be available. If this was part of a subset to download, remove the resource id from the subset. ## Offline localHub usage There is a flag in the constructor `localHub=TRUE` to use only show previously downloaded resources. If you are using this in an offline setting, please also see the [BiocManager vignette on offline use](https://cran.r-project.org/web/packages/BiocManager/vignettes/BiocManager.html#offline-use). The Hubs use BiocManager::version to filter resources appropriate for your current version of R/Bioconductor; therefore the offline usage of BiocManager needs to be established. The error that occurs that likely needs offline set up will look similar to the following: ``` ExperimentHub(localHub=TRUE) Error: failed to connect to local data base database: ‘/home/user/.cache/R/ExperimentHub/24dc84dbf615e_experimenthub.sqlite3’ reason: invalid version specification ‘Bioconductor version cannot be validated; no internet connection?’` ``` # Group Hub/Cache Access The situation may occur where a hub is desired to be shared across multiple users on a system. This presents permissions errors. To allow access to multiple users create a group that the users belong to and that the underlying hub cache belongs too. Permissions of potentially two files need to be altered depending on what you would like individuals to be able to accomplish with the hub. A read-only hub will require manual manipulatios of the file BiocFileCache.sqlite.LOCK so that the group permissions are `g+rw`. To allow users to download files to the shared hub, both the BiocFileCache.sqlite.LOCK file and the BiocFileCache.sqlite file will need group permissions to `g+rw`. Please google how to create a user group for your system of interest. To find the location of the hub cache to be able to change the group and file permissions, you may run the following in R `ah = AnnotationHub(); hubCache(ah)`. For quick reference in linux you will use `chown currentuser:newgroup` to change the group and `chmod` to change the file permissions: `chmod 660` or `chmod g+rw` should accomplish the correct permissions. # Lock file Troubleshooting Two issues have been commonly reported regarding the lock file. ## Permissions There could be permission ERROR regarding group and public access. See the previous `Group Cache Access` section. ## Cannot lock file / no lock available This is an issue with filelock on particular systems. Particular partitions and non standard file systems may not support filelock. The solution is to use a different section of the system to create the cache. The easiest way to define a new cache location is by using environment variables. In R: `Sys.setenv(BFC_CACHE=)` Alternatively, you can set an environment variable globally to avoid having to set uniquely in each R session. Please google for specific instructions for setting environment variables globally for your particular OS system. Other common filelock implemented packages that have specific environment variables to control location are: * BiocFileCache: BFC_CACHE * ExperimentHub: EXPERIMENT_HUB_CACHE * AnnotationHub: ANNOTATION_HUB_CACHE * biomaRt: BIOMART_CACHE # Default Caching Location Update As of AnnotationHub version > 2.23.2, the default caching location has changed. The default cache is now controlled by the function `tools::R_user_dir` instead of `rappdirs::user_cache_dir`. Users who have utilized the default AnnotationHub location, to continue using the created cache, must move the cache and its files to the new default location, create a system environment variable to point to the old location, or delete and start a new cache. ## Option 1: Move cache and files to new location The following steps can be used to move the files to the new location: 1. Determine the old location by running the following in R `rappdirs::user_cache_dir(appname="AnnotationHub")` 2. Determine the new location by running the following in R `tools::R_user_dir("AnnotationHub", which="cache")` 3. Move the files to the new location. You can do this manually or do the following steps in R. Remember if you have a lot of cached files, this may take awhile. ```{r, eval=FALSE} # make sure you have permissions on the cache/files # use at own risk moveFiles<-function(package){ olddir <- path.expand(rappdirs::user_cache_dir(appname=package)) newdir <- tools::R_user_dir(package, which="cache") dir.create(path=newdir, recursive=TRUE) files <- list.files(olddir, full.names =TRUE) moveres <- vapply(files, FUN=function(fl){ filename = basename(fl) newname = file.path(newdir, filename) file.rename(fl, newname) }, FUN.VALUE = logical(1)) if(all(moveres)) unlink(olddir, recursive=TRUE) } package="AnnotationHub" moveFiles(package) ``` ## Option 2: Create a system environment variable A user may set the system environment variable `ANNOTATION_HUB_CACHE` to control the default location of the cache. Setting system environment variables can vary depending on the operating system; we suggest using google to find appropriate instructures per your operating system. You will want to set the variable to the results of running the following in R: ```{r, eval=FALSE} path.expand(rappdirs::user_cache_dir(appname="AnnotationHub")) ``` **NOTE:** R has `Sys.setenv` however that will only set the variable for that R session. It would not be available or recognized in future R sessions. It is important to set the variable as a global user-wide or system-wide environment variable so it persists in all future logins to your system. ## Option 3: Delete the old cache Lastly, if a user does not care about the already existing default cache, the old location may be deleted to move forward with the new default location. This option should be used with caution. Once deleted, old cached resources will no longer be available and have to be re-downloaded. One can do this manually by navigating to the location indicated in the ERROR message as `Problematic cache:` and deleting the folder and all its content. The following can be done to delete through R code: **CAUTION** This will remove the old cache and all downloaded resources. All resources will have to be re-downloaded after executing this code. ```{r, eval=FALSE} library(AnnotationHub) package = "AnnotationHub" oldcache = path.expand(rappdirs::user_cache_dir(appname=package)) setAnnotationHubOption("CACHE", oldcache) ah = AnnotationHub(localHub=TRUE) ## removes old location and all resources removeCache(ah, ask=FALSE) ## create the new default caching location newcache = tools::R_user_dir(package, which="cache") setAnnotationHubOption("CACHE", newcache) ah = AnnotationHub() ``` [Bioconductor Support Site]: https://support.bioconductor.org/ [AnnotationHub]: https://bioconductor.org/packages/AnnotationHub/ [ExperimentHub]: https://bioconductor.org/packages/ExperimentHub/ [BiocFileCache]: https://bioconductor.org/packages/BiocFileCache/ [dplyr]: https://www.rdocumentation.org/packages/dplyr/versions/0.7.8