Download a S3 Folder using React Native and redux-observable
I recently had to provide the ability to download directories stored in AWS S3 in a React Native application. Since there is no way to do it out-of-the box using the official AWS API, I had to come up with something custom that I actually find useful to share here. I’m a convinced user of redux-observable (and RxJS in general) and this turned out to be an incredibly valuable asset to achieve this (as it is for a whole lot of stuff :D).
To put this feature into context, I will setup a very simple application, allowing you to download some albums containing pictures. Each album will also contain a cover that will be its visible part. When successfully downloaded, the albums will be visible through their covers. We will also display the progress of the currently downloading album.
let’s start by using create-react-native-app to bootstrap our codebase. Go ahead and create the app through:
We’e going to use some native features, so you want to eject your app through:
after the wizardry is done, go ahead and install the packages we will use throughout the tutorial:
Before writing some code, let’s define what is going to be the structure of the project. There are going to be 3 main areas composing the application:
- The fetch and display of available albums on S3 (available-albums)
- The download of albums (album-download)
- The display of downloaded albums (my-albums)
every area will have its own reducer to keep track of its state, as well as its related set of actions. We’re also going to use redux-observable epics to handle the side effects such as fetching and downloading. If you are new to this library, I encourage you to first take a look at the documentation. If you’re already familiar with RxJS, this should feel pretty natural to you.
The albums I will be using for this tutorial are made of photos taken from https://unsplash.com/. I have created three albums grouped by categories: animals, portraits, and landscapes. You can access them here:
Just download them, unzip and upload the whole folders on your S3 Bucket of choice. Inside each album, I have created a cover, that is an edition of an existing picture of the album, optimized for standalone display. This is going to be the publicly visible part of the album. For that reason, you will also have to make them public within your bucket. To do that, you can just go ahead and right click on the file on the AWS S3 Console, and select ‘Make public’:
Of course, to make things easier, you could just make everything public. But in my case, I wanted to prevent anyone from downloading the content of the directories that are not a visible part.
Let’s now dive into some code:
Displaying the Album Covers
The first thing we want to do, is to display the albums through their covers and allow the user to tap on them to trigger their download. I will start by writing a module, along with a function responsible for the fetch of the album files on AWS S3. I’ll put the module into a available-albums directory in my codebase:
In your actual codebase, you wouldn’t probably store the AWS credentials here, but maybe using something like: react-native-config
fetchFiles function makes use of the
listObjects API of the S3 SDK. I start by translating the resulting Promise into an Observable since it’s going to be our bread und butter for all our manipulations. What I’m doing after this, is mapping in order to get the information we need for every file and nothing else. In this case, I will only need the key (the absolute name within its Bucket) and its length.
The issue with the S3 SDK, is that it has no notion of directory per se. The content of the Observable will be an array that will look like this:
So the next step is to group the files by their album, which we’re going to do with the help of the
groupBy function from lodash. We’re also going to enhance the structure with the public URL of the cover of each album:
The new structure will now look like:
We can now expose a function putting these together:
Let’s now define the actions as well as the reducer:
Nothing really fancy here. In real world application, you’d probably want to keep track of the fetching state to be able to display a spinner, but we don’t need that for our example.
Next, let’s write the epic that will be responsible for triggering the available albums fetch. It’s going to react to the action of type FETCH_AVAILABLE_ALBUMS and will respond with the result of the
fetchAvailableAlbums function we have written:
our epic is a function taking an input action as an Observable, the store and a set of dependencies that can be injected during the store setup. I’m not using the store in that example so I defined the parameter as ‘_’. I’m interested in one dependency only which is the module I have previously created, that I will inject later on.
The principle of an epic is pretty simple: start with an input stream of actions, handle your async side effects, and finish by returning an output stream of actions. We’re doing it by taking advantage of the RxJS lettable operators, and the pipe method. the ofType function provided by redux-observable is a syntactic sugar to filter the actions by their type. We use the switchMap operator to output an Observable that is going to encapsulate the result of the
fetchAlbumFiles function, through the
albumsFetched action creator. This is going to trigger the state change in our reducer.
We’re now equipped to actually display something on the screen. First, I’m going to write two simple, stateless components that are going to be used for both our screens:
The first one is a ScrollView with some margin that is going to host the album covers we want to display. The second is the actually visible cover. We want it to be tappable, so we use a TouchableWithoutFeedback wrapper. The cover is hosted inside an ImageBackground with a fixed height of 100 and takes the whole screen width. We also want to be able to display something on top of it (more on that later), hence the children prop.
Now let’s create the
We connect the component to the
availableAlbums state, and the
fetchAvailableAction. The rendering consists of looping over the availableAlbum keys (the album names), and the display of an AlbumCover, with the uri prop being the corresponding coverUrl for each one of them.
The last step before visualizing the result is to wire all the stuff up with some store boilerplate. Let’s do that through a
store.js file that we’ll write in the
We use the
createEpicMiddleware function provided by redux-observable to register the epics used in the application, along with the dependencies. The epics are combined all together through the
combineEpics function. finally, we declare our epicMiddleware as a middleware when creating the store through
Let’s now put this to work in our actual application. Open up
App.js in the root directory and edit it like so:
If you’re wondering what the SafeAreaView is, that’s here to help us stay in safe bounds of the iPhone X without the status bar overlapping with our stuff.
Download the Albums
Ok, now that we’re able to display the albums from our S3 Bucket, we want to be able to download them on the device.
Let’s first write a module exposing a download function:
This looks a little bit straightforward so let’s go over this a little bit. What we want to do, is to download all the files contained in the album object sequentially. This is what the downloadSequentially function is for. It takes the files in input, and a callback that will be called every time a file has finished downloading. We use RNFetchBlob to do the actual file downloads, and store everything in the Document directory. We’re basically chaining every download resulting promise together, to make sure that the execution is sequential. Finally, we wrap the whole thing into an Observable to make it RxJS friendly.
The download function uses this function and outputs two streams: the final download result, as well as each file download result. The file download result stream takes its source from a RxJS Subject, which is finally transformed into an Observable before returned along with the download result stream.
After having digested this, go ahead and create the album-download related actions set:
now, let’s create the corresponding reducer:
There is a bit more stuff here than in our previous reducer. The basic idea here is to keep track of the download progress. when an action of type
DOWNLOAD_ALBUM is received, we set the progress to 0. We also calculate the totalLength by summing up the length of each file in the payload (the album). We also need the coverUrl as it’s going to be displayed as well.
ALBUM_FILE_DOWNLOADED, we’re going to update the progress, which is going to be incremented by the fraction of the file length over the total length.
We can to write the epic that is going to be triggered by an album download. But before we do that, we need to add a function to the
AlbumsRepository enabling us to get the signed download links from S3. As we mentioned before, we don’t want all file to be publicly accessible like the covers are.
Let’s edit the file and add:
The exported withSignedDownloadUrls function enhances the album structure by adding a downloadUrl to each file. This is done through the getSignedUrl function available in the S3 SDK.
You are probably wondering why we decide to retrieve the signed URLs now, and not already when we retrieve the album files. We’re doing so because signing an URL is a temporary process, meaning that it’s going to expire at some point (15min by default). Of course, you can increase this value through the S3 console, but that’s going to be in detriment of security.
We’re now fully equipped to create the
The process is fairly similar as our previous epic. We use the
withSignedDownloadUrls to enhance the album structure. We then combine the progress streams using the mergeMap operator. Each one of them will result in a trigger of an
ALBUM_FILE_DOWNLOADED action. The download result stream will itself turn into an
ALBUM_DOWNLOADED at completion. finally, the output stream is simply the combination of all these streams together.
At some point, we probably also want to display the downloaded albums, which means that we need to keep track of them.
Let’s write our last reducer for that. We’re going to place it into a new folder called my-albums:
Let’s now create some UI for that. Go ahead and create a file
MyAlbums.js in the same my-albums folder. It’s going allow us to display the downloaded albums, as well as visualizing the currently downloading one and its progress:
We use the AlbumShelf to display the downloaded albums, as well as the one currently downloading if there’s one (albumDownload state not equal to the
initialAlbumDownload). You know get why we allowed the AlbumCover to host a children prop. We’re using a ProgressOverlay functional component to display the progress of the current download (through the react-native-progress package).
Ok, let’s now wire this up to the rest of the application. Open up
App.js and add a new tab from the
Now, edit the store.js file to add the missing bits:
And finally, let’s wire up the downloadAlbum action to
And that should be it !
Of course, we must now do something useful with the downloaded files, but that’s beyond the scope of this tutorial. Also, in a real world application, we would typically handle the network errors, allow the cancel of downloads etc… but with redux-observable in place, that’s really a breeze.
I hope this article has been useful to you, and if so, please go ahead and share it.
Thanks Alex Iby, Ariel Lustre, Ayo Ogunseide, Ben Blennerhassett, Beth Solano, Christopher Campbell, Enis Yavuz, Erik Lucatero, Joe Gardner, Lucas Sankey, michael, Stacey Rozells, Laura College, Alan Emery, Alejandro Contreras, Baptist Standaert, Boris Smokrovic, Cara Fuller, David Clode, Frida Bredesen, Jean Gaspar de Alba, Linnea Sandbakk, Paul, Ricky Kharawala, Ryan Walton, Wynand van Poortvliet, Abigail Keenan, Chang Qing, Dan Grinwis, Diego Jimenez, Ghost Presenter, Hugues de BUYER-MIMEURE, Jakub Kriz, Johny Goerend, Mark Tuzman, Nikkhita Singhal, Preston Pownell and Sebastian Pichler for the amazing pictures.AWS, React Native, RxJS. Bookmark the permalink.