hexV

hexV is a little tool that I've been working on for a while.
It's a tiny viewer that shows the content of any file in hexadecimal.
Something like this:

I first thought about making hexV while I was working on another project that involved parsing and rendering large images without using any official libraries like libpng. I needed something that I could use to learn and understand how images are structured and stored depending on the type (PNG, JPEG, BMP...), and also that'd help me debug the parsers I wrote for that project.

hexV did the trick. Instead of having to go through endless console output to chase bugs, I could simply open the processed image with hexV and compare it to the values read by the parsers. I'm pretty sure that using hexV saved me a ton of debugging time.

The first version of hexV had builds for both windows and linux, but I recently added some deep changes to the linux version so I removed the windows build until I can make some time to add those changes to it.

I have some interesting features in mind, like file type detection, that I'd like to add sometime soon. 

Oh, and this is an open-source project, so if you find hexV interesting and feel like collaborating, take a look at the issues and hack away!

Read More

Telescope 1.0

It's live!

Telescope 1.0 is up and running, and it's thanks to an incredible group of people who worked really hard to make it happen.

For this final release, I wanted to add some nice features to Telescope, but because of the situation we're going through right now, I found sitting down and working on stuff more difficult than usual.

I helped with some minor issues and some production related stuff, but I couldn't finish the features I was working on, so they were not included in this big release.

This is not the end of the road. I know I'll eventually add these features I was working on to Telescope, and I know I'll be helping with many other things, things we haven't even thought of yet.

Anyway, I really liked this project, and I'm sure I'll keep working on it for a long time.

What about you? Would you like to contribute? Join us, and tell us about your experience with Open Source!

Read More

Telescope: 0.9

We're getting close to 1.0 release, and it's time to add important features so we can get a robust, versatile and also easy to use Telescope 1.0.

For this release I worked on some interesting issues:

Unreachable feed list
Telescope gets a list of feeds from a server that sometimes it's not available, and when that happens it causes Telescope to crash. I added a small fix and now if the server is not available, Telescope will keep trying to get the list from the server with little delays in between attempts.

Nginx
I also worked with @c3ho to make some changes to our reverse proxy, and now Telescope uses nginx to serve static content and cache its endpoints. The interesting part is that we had to figure out the right settings to cache only certain endpoints to avoid security issues. Check out some other interesting changes he made to our nginx settings.

Persistent Storage
I liked this one. It was a tiny change, but before this addition Telescope had to process all our feeds (and posts) every time a new PR was merged to master. Now we have persistent storage, so our processed content gets stored on disk and fresh Telescope deployments can use that stored content to show fully updated feeds and posts after every merge.

For 1.0
I also wanted to add something similar to what I did for Elasticsearch, but for Redis. Currently, Telescope waits for Elasticsearch to be ready before start processing feeds, but it doesn't wait for Redis. This could cause data loss or even unexpected crashes, so I'll be working on this for 1.0.

Read More

Telescope: Elasticsearch and Automatic Deployment (Part 2)

I mentioned in my previous post that I started working on adding Elasticsearch and automatic deployment to Telescope, and because of some minor problems, I couldn't include them in our 0.7 release.
But yes, I managed to fix all the problems for 0.8, and now they're part of Telescope!

For 0.8, I also added a new nested query for GraphQL. This new query returns information about a feed and all the posts related to it, all in one request. I think there will be some cases once we fully implement a search functionality that will benefit from this new query.

I also had the chance to briefly work with @Silvyre on adding graphql-passport to Telescope. It turned out to be a bit trickier than we expected, but @Silvyre was determined to get it ready for 0.8, and thanks to his big effort graphql-passport is now part of Telescope.

Elasticsearch has become a core part of the project, and it was very important to add documentation about installation and deployment for this new piece. The main idea was to avoid blocking developers that are not familiar with Elasticsearch, so I added new documentation on the different ways to install and use Elasticsearch locally, and how to test it.

@c3ho and I spent some time working on our nginx settings so we could get nginx to serve Telescope's static content and also enable caching for our endpoints (for this issue). We managed to get those 2 pieces to work, but since I still wanted to review a couple of things I thought it was better to leave this for 0.9.

Read More

Telescope: Elasticsearch and Automatic Deployment

For the 0.7 release, I've been working on a couple of issues: adding text indexing and automatic deployment.


Text indexing
Adding text indexing to Telescope was interesting. After checking out some options, Elasticsearch (a distributed, RESTful search and analytics engine) was the tool that we decided to use to implement text indexing. I added a new container running Elasticsearch to our workflow that will be waiting for requests to index blogposts or search for already indexed blogposts that contain a passed string.

Surprisingly, the main problem I had to deal with was not functionality or interactivity with Elasticsearch, but mocking it. Elasticsearch works great with Telescope, but a fourth of our tests fail due connectivity problems (Elasticsearch is down when I run tests, so the client complains). I did a bit of research to find out about use an Elasticsearch client in mock mode to mimic what we do with redis, but from what I've seen it's not as easy as I thought it'd be and it might involve adding some Java to the mix, which is something that we don't really want for Telescope.

I'm still working on a solution (I'll have a PR ready soon), but I think I'll wrap the Elasticsearch pieces in our code in conditionals that will only run when an environment variable is set to a specific value.


Automatic deployment
For 0.7, we agreed to add simple auto deployment. The ultimate goal is to have blue/green deployment, but since that'll probably involve deep changes to our current docker settings I think it's a good idea to take this first step and then try to build on it in future releases.

To implement automatic deployment, I added a small server that waits for requests from GitHub to shutdown, reclone, and redeploy Telescope whenever a pull request is merged to master. For this, we're using GitHub webhooks (Calvin I worked on this part together).

I got this piece to work on my machine, but I still have a small issue with our .env file. Hopefully I'll be able to fix it soon and add this to our staging server soon.


Other issues
I also did a bit of work to have persistent storage with redis and adding some more filters and nested queries to our GraphQL. I think these 2 will very likely be included in the 0.8 release.

Read More

Telescope: Nginx and Let's encrypt

So, Telescope is getting closer to become the successor for PlanetCDOT. If you still don't know what Telescope is (shame on you), check this out.

Nginx & Let's Encrypt
I spent this week researching nginx to add a reverse proxy to our staging box so we could add some cool features. Also, I worked with another contributor (@miggs125) to add Let's Encrypt so we can use SSL.

After dealing with some configuration issues, we managed to get it to work, and once we merge the necessary changes, we'll have Telescope using SSL. I think I've probably said it too many times, but once again thanks to Telescope, I had the change to learn new things: I got to learn how to setup a reverse proxy and how to add SSL to a site.

Feed endpoints
As I mentioned in previous posts, Telescope has a REST API and GraphQL. The endpoints to serve posts were implemented some time ago, but we were still lacking a way to serve feeds to our frontend. Since I collaborated adding the endpoints for posts, I thought it'd be a good idea to work on the same issue but for feeds. I added endpoints to request feeds and one feed, along with some tests for those endpoints.

GraphQL filters
I also did some work for GraphQL. Another contributor (@c3ho) is working on adding filters to our Apollo server, so our queries will be more flexible and efficient. It's a very tricky task, so I offered to help since I worked on the initial addition of Apollo to Telescope. We've made some progress and, to be honest, if he manages to include all the functionalities we want (taking advantage of some Apollo's features), the improvement will be significant. Seriously, really interesting stuff.

Read More

Back to Telescope

It's live!

Yes, Telescope is live, only its development version though, but it is live, and you can check it out here. If you have no clue about what Telescope is, go check my previous posts to learn about it. Go, I'll wait.

Ready? So here we are again, trying to add more features to Telescope, learning new things, and getting excited again!

It's becoming a very intense project, so I'll go right away to the things I've been involved with:

- GraphQL

 GraphQL is a query and manipulation language for APIs and a runtime for fulfilling queries with existing data. Adding it to Telescope wasn't easy, but it's done, and now we can take advantage of its features.


- Hashing and encoding IDs

This one was a bit trickier. It was my first time doing this so I had to get a bit familiar with crypto, a wrapper for OpenSSL cryptographic functions, but once I understood what had to be done (and with some extra help from others involved in the project), I managed to integrate it with what we had.


- Minikube and Kubernetes

Minikube and Kubernetes additions to Telescope are still WIP, but we've made some progress with them. I'm working with another contributor to run Telescope using a Kubernetes cluster, but due to our lack of experience, we're trying first with minikube. I think we're getting there, we did some testing and we hit some walls that you're supposed to hit when you're learning this stuff. If everything goes well, and our guesses are correct, I think we'll be able to get it run in a week or two.


- Deployment

This is another very interesting piece. Right now Telescope is running on port 80, but it'd be nice (and professional) to use SSL, right? Well, that's what I'm going to be doing soon. I just started researching Nginx to see how it works and how it can be used with docker-compose (which is what we using for our staging server), and once Ngingx is added to our docker-compose file, we'll try to convince Let's Encrypt that we're trustworthy.

That's pretty much what's happening right now, Telescope keeps growing at a pace that's almost hard to keep up with, but what we're getting out of it in terms of experience and knowledge is simply awesome.

Oh, and we're adding the fanciest toy in the store right now, Gatsbyjs.

Stay tuned!

Read More