Tuesday, March 24, 2020

Telescope: Elasticsearch and Automatic Deployment (Part 2)

I mentioned in my previous post that I started working on adding Elasticsearch and automatic deployment to Telescope, and because of some minor problems, I couldn't include them in our 0.7 release.
But yes, I managed to fix all the problems for 0.8, and now they're part of Telescope!

For 0.8, I also added a new nested query for GraphQL. This new query returns information about a feed and all the posts related to it, all in one request. I think there will be some cases once we fully implement a search functionality that will benefit from this new query.

I also had the chance to briefly work with @Silvyre on adding graphql-passport to Telescope. It turned out to be a bit trickier than we expected, but @Silvyre was determined to get it ready for 0.8, and thanks to his big effort graphql-passport is now part of Telescope.

Elasticsearch has become a core part of the project, and it was very important to add documentation about installation and deployment for this new piece. The main idea was to avoid blocking developers that are not familiar with Elasticsearch, so I added new documentation on the different ways to install and use Elasticsearch locally, and how to test it.

@c3ho and I spent some time working on our nginx settings so we could get nginx to serve Telescope's static content and also enable caching for our endpoints (for this issue). We managed to get those 2 pieces to work, but since I still wanted to review a couple of things I thought it was better to leave this for 0.9.


Share:

Sunday, March 1, 2020

Telescope: Elasticsearch and Automatic Deployment

For the 0.7 release, I've been working on a couple of issues: adding text indexing and automatic deployment.


Text indexing
Adding text indexing to Telescope was interesting. After checking out some options, Elasticsearch (a distributed, RESTful search and analytics engine) was the tool that we decided to use to implement text indexing. I added a new container running Elasticsearch to our workflow that will be waiting for requests to index blogposts or search for already indexed blogposts that contain a passed string.

Surprisingly, the main problem I had to deal with was not functionality or interactivity with Elasticsearch, but mocking it. Elasticsearch works great with Telescope, but a fourth of our tests fail due connectivity problems (Elasticsearch is down when I run tests, so the client complains). I did a bit of research to find out about use an Elasticsearch client in mock mode to mimic what we do with redis, but from what I've seen it's not as easy as I thought it'd be and it might involve adding some Java to the mix, which is something that we don't really want for Telescope.

I'm still working on a solution (I'll have a PR ready soon), but I think I'll wrap the Elasticsearch pieces in our code in conditionals that will only run when an environment variable is set to a specific value.


Automatic deployment
For 0.7, we agreed to add simple auto deployment. The ultimate goal is to have blue/green deployment, but since that'll probably involve deep changes to our current docker settings I think it's a good idea to take this first step and then try to build on it in future releases.

To implement automatic deployment, I added a small server that waits for requests from GitHub to shutdown, reclone, and redeploy Telescope whenever a pull request is merged to master. For this, we're using GitHub webhooks (Calvin I worked on this part together).

I got this piece to work on my machine, but I still have a small issue with our .env file. Hopefully I'll be able to fix it soon and add this to our staging server soon.


Other issues
I also did a bit of work to have persistent storage with redis and adding some more filters and nested queries to our GraphQL. I think these 2 will very likely be included in the 0.8 release.
Share: