Sunday, April 19, 2020

Telescope 1.0

It's live!

Telescope 1.0 is up and running, and it's thanks to an incredible group of people who worked really hard to make it happen.

For this final release, I wanted to add some nice features to Telescope, but because of the situation we're going through right now, I found sitting down and working on stuff more difficult than usual.

I helped with some minor issues and some production related stuff, but I couldn't finish the features I was working on, so they were not included in this big release.

This is not the end of the road. I know I'll eventually add these features I was working on to Telescope, and I know I'll be helping with many other things, things we haven't even thought of yet.

Anyway, I really liked this project, and I'm sure I'll keep working on it for a long time.

What about you? Would you like to contribute? Join us, and tell us about your experience with Open Source!
Share:

Sunday, April 5, 2020

Telescope: 0.9

We're getting close to 1.0 release, and it's time to add important features so we can get a robust, versatile and also easy to use Telescope 1.0.

For this release I worked on some interesting issues:

Unreachable feed list
Telescope gets a list of feeds from a server that sometimes it's not available, and when that happens it causes Telescope to crash. I added a small fix and now if the server is not available, Telescope will keep trying to get the list from the server with little delays in between attempts.

Nginx
I also worked with @c3ho to make some changes to our reverse proxy, and now Telescope uses nginx to serve static content and cache its endpoints. The interesting part is that we had to figure out the right settings to cache only certain endpoints to avoid security issues. Check out some other interesting changes he made to our nginx settings.

Persistent Storage
I liked this one. It was a tiny change, but before this addition Telescope had to process all our feeds (and posts) every time a new PR was merged to master. Now we have persistent storage, so our processed content gets stored on disk and fresh Telescope deployments can use that stored content to show fully updated feeds and posts after every merge.

For 1.0
I also wanted to add something similar to what I did for Elasticsearch, but for Redis. Currently, Telescope waits for Elasticsearch to be ready before start processing feeds, but it doesn't wait for Redis. This could cause data loss or even unexpected crashes, so I'll be working on this for 1.0.


Share:

Tuesday, March 24, 2020

Telescope: Elasticsearch and Automatic Deployment (Part 2)

I mentioned in my previous post that I started working on adding Elasticsearch and automatic deployment to Telescope, and because of some minor problems, I couldn't include them in our 0.7 release.
But yes, I managed to fix all the problems for 0.8, and now they're part of Telescope!

For 0.8, I also added a new nested query for GraphQL. This new query returns information about a feed and all the posts related to it, all in one request. I think there will be some cases once we fully implement a search functionality that will benefit from this new query.

I also had the chance to briefly work with @Silvyre on adding graphql-passport to Telescope. It turned out to be a bit trickier than we expected, but @Silvyre was determined to get it ready for 0.8, and thanks to his big effort graphql-passport is now part of Telescope.

Elasticsearch has become a core part of the project, and it was very important to add documentation about installation and deployment for this new piece. The main idea was to avoid blocking developers that are not familiar with Elasticsearch, so I added new documentation on the different ways to install and use Elasticsearch locally, and how to test it.

@c3ho and I spent some time working on our nginx settings so we could get nginx to serve Telescope's static content and also enable caching for our endpoints (for this issue). We managed to get those 2 pieces to work, but since I still wanted to review a couple of things I thought it was better to leave this for 0.9.


Share:

Sunday, March 1, 2020

Telescope: Elasticsearch and Automatic Deployment

For the 0.7 release, I've been working on a couple of issues: adding text indexing and automatic deployment.


Text indexing
Adding text indexing to Telescope was interesting. After checking out some options, Elasticsearch (a distributed, RESTful search and analytics engine) was the tool that we decided to use to implement text indexing. I added a new container running Elasticsearch to our workflow that will be waiting for requests to index blogposts or search for already indexed blogposts that contain a passed string.

Surprisingly, the main problem I had to deal with was not functionality or interactivity with Elasticsearch, but mocking it. Elasticsearch works great with Telescope, but a fourth of our tests fail due connectivity problems (Elasticsearch is down when I run tests, so the client complains). I did a bit of research to find out about use an Elasticsearch client in mock mode to mimic what we do with redis, but from what I've seen it's not as easy as I thought it'd be and it might involve adding some Java to the mix, which is something that we don't really want for Telescope.

I'm still working on a solution (I'll have a PR ready soon), but I think I'll wrap the Elasticsearch pieces in our code in conditionals that will only run when an environment variable is set to a specific value.


Automatic deployment
For 0.7, we agreed to add simple auto deployment. The ultimate goal is to have blue/green deployment, but since that'll probably involve deep changes to our current docker settings I think it's a good idea to take this first step and then try to build on it in future releases.

To implement automatic deployment, I added a small server that waits for requests from GitHub to shutdown, reclone, and redeploy Telescope whenever a pull request is merged to master. For this, we're using GitHub webhooks (Calvin I worked on this part together).

I got this piece to work on my machine, but I still have a small issue with our .env file. Hopefully I'll be able to fix it soon and add this to our staging server soon.


Other issues
I also did a bit of work to have persistent storage with redis and adding some more filters and nested queries to our GraphQL. I think these 2 will very likely be included in the 0.8 release.
Share:

Saturday, February 1, 2020

Telescope: Nginx and Let's encrypt

So, Telescope is getting closer to become the successor for PlanetCDOT. If you still don't know what Telescope is (shame on you), check this out.

Nginx & Let's Encrypt
I spent this week researching nginx to add a reverse proxy to our staging box so we could add some cool features. Also, I worked with another contributor (@miggs125) to add Let's Encrypt so we can use SSL.

After dealing with some configuration issues, we managed to get it to work, and once we merge the necessary changes, we'll have Telescope using SSL. I think I've probably said it too many times, but once again thanks to Telescope, I had the change to learn new things: I got to learn how to setup a reverse proxy and how to add SSL to a site.

Feed endpoints
As I mentioned in previous posts, Telescope has a REST API and GraphQL. The endpoints to serve posts were implemented some time ago, but we were still lacking a way to serve feeds to our frontend. Since I collaborated adding the endpoints for posts, I thought it'd be a good idea to work on the same issue but for feeds. I added endpoints to request feeds and one feed, along with some tests for those endpoints.

GraphQL filters
I also did some work for GraphQL. Another contributor (@c3ho) is working on adding filters to our Apollo server, so our queries will be more flexible and efficient. It's a very tricky task, so I offered to help since I worked on the initial addition of Apollo to Telescope. We've made some progress and, to be honest, if he manages to include all the functionalities we want (taking advantage of some Apollo's features), the improvement will be significant. Seriously, really interesting stuff.


Share:

Friday, January 24, 2020

Back to Telescope

It's live!

Yes, Telescope is live, only its development version though, but it is live, and you can check it out here. If you have no clue about what Telescope is, go check my previous posts to learn about it. Go, I'll wait.

Ready? So here we are again, trying to add more features to Telescope, learning new things, and getting excited again!

It's becoming a very intense project, so I'll go right away to the things I've been involved with:

- GraphQL

 GraphQL is a query and manipulation language for APIs and a runtime for fulfilling queries with existing data. Adding it to Telescope wasn't easy, but it's done, and now we can take advantage of its features.


- Hashing and encoding IDs

This one was a bit trickier. It was my first time doing this so I had to get a bit familiar with crypto, a wrapper for OpenSSL cryptographic functions, but once I understood what had to be done (and with some extra help from others involved in the project), I managed to integrate it with what we had.


- Minikube and Kubernetes

Minikube and Kubernetes additions to Telescope are still WIP, but we've made some progress with them. I'm working with another contributor to run Telescope using a Kubernetes cluster, but due to our lack of experience, we're trying first with minikube. I think we're getting there, we did some testing and we hit some walls that you're supposed to hit when you're learning this stuff. If everything goes well, and our guesses are correct, I think we'll be able to get it run in a week or two.


- Deployment

This is another very interesting piece. Right now Telescope is running on port 80, but it'd be nice (and professional) to use SSL, right? Well, that's what I'm going to be doing soon. I just started researching Nginx to see how it works and how it can be used with docker-compose (which is what we using for our staging server), and once Ngingx is added to our docker-compose file, we'll try to convince Let's Encrypt that we're trustworthy.

That's pretty much what's happening right now, Telescope keeps growing at a pace that's almost hard to keep up with, but what we're getting out of it in terms of experience and knowledge is simply awesome.

Oh, and we're adding the fanciest toy in the store right now, Gatsbyjs.

Stay tuned!
Share:

Thursday, January 2, 2020

KDE Plasma & ssh keys

If you're a Linux user, and the desktop environment of your choice is Gnome, you're probably used to letting Gnome Keyring SSH Agent handle your ssh keys. You just log in, your ssh keys stored in your ~/.ssh folder get loaded in memory, and then you can use them not only in terminals but with any process that requires ssh authorization.

Unfortunately, KDE Plasma doesn't have that feature out of the box, so it needs a bit of tweaking to get the same behaviour.

Let's make some changes to Kwallet and add some scripts to start our ssh-agent and load our keys:


Kwallet
Launch KDE Wallet Configuration and make sure the KDE wallet subsystem is enabled.
Launch Kwallet Manager and create a new wallet if necessary and set a passphrase for it.


Scripts
Now we need to create some scripts to start the ssh-agent on startup, add all the keys, and stop it on shutdown. For this, it's necessary to have the package ksshaskpass installed.

KDE has a designated folder for scripts that will be executed at login but before launching Plasma.

Folder: ~/.config/plasma-workspace/env

In this folder, we need to create a script to start the ssh-agent. Let's call it ssh-agent-startup.sh.

#!/bin/bash

[ -n "$SSH_AGENT_PID" ] || eval "$(ssh-agent -s)"


Also, KDE uses another folder for scripts at login.

Folder: ~/.config/autostart-scripts

Let's add a script to load all our ssh keys. We'll call our script ssh-add.sh.

#!/bin/bash

export SSH_ASKPASS=/usr/bin/ksshaskpass

ssh-add $HOME/.ssh/my_ssh_key1 $HOME/.ssh/my_ssh_key2 $HOME/.ssh/my_ssh_key3...


And finally, let's add a script to stop our ssh-agent at shutdown.

Folder: ~/config/plasma-workspace/shutdown

Our script will be ssh-agent-shutdown.sh.

#!/bin/bash

[ -z "$SSH_AGENT_PID" ] || eval "$(ssh-agent -k)"


Don't forget to mark the scripts as executables:

chmod +x file/to/mark/as/executable


And that's it. After rebooting, the system will prompt you to enter your keys' passphrases, and if everything went well, you should be able to use your keys with any process that needs ssh authorization.

Share: