Showing posts with label docker. Show all posts
Showing posts with label docker. Show all posts

Sunday, March 1, 2020

Telescope: Elasticsearch and Automatic Deployment

For the 0.7 release, I've been working on a couple of issues: adding text indexing and automatic deployment.


Text indexing
Adding text indexing to Telescope was interesting. After checking out some options, Elasticsearch (a distributed, RESTful search and analytics engine) was the tool that we decided to use to implement text indexing. I added a new container running Elasticsearch to our workflow that will be waiting for requests to index blogposts or search for already indexed blogposts that contain a passed string.

Surprisingly, the main problem I had to deal with was not functionality or interactivity with Elasticsearch, but mocking it. Elasticsearch works great with Telescope, but a fourth of our tests fail due connectivity problems (Elasticsearch is down when I run tests, so the client complains). I did a bit of research to find out about use an Elasticsearch client in mock mode to mimic what we do with redis, but from what I've seen it's not as easy as I thought it'd be and it might involve adding some Java to the mix, which is something that we don't really want for Telescope.

I'm still working on a solution (I'll have a PR ready soon), but I think I'll wrap the Elasticsearch pieces in our code in conditionals that will only run when an environment variable is set to a specific value.


Automatic deployment
For 0.7, we agreed to add simple auto deployment. The ultimate goal is to have blue/green deployment, but since that'll probably involve deep changes to our current docker settings I think it's a good idea to take this first step and then try to build on it in future releases.

To implement automatic deployment, I added a small server that waits for requests from GitHub to shutdown, reclone, and redeploy Telescope whenever a pull request is merged to master. For this, we're using GitHub webhooks (Calvin I worked on this part together).

I got this piece to work on my machine, but I still have a small issue with our .env file. Hopefully I'll be able to fix it soon and add this to our staging server soon.


Other issues
I also did a bit of work to have persistent storage with redis and adding some more filters and nested queries to our GraphQL. I think these 2 will very likely be included in the 0.8 release.
Share:

Friday, January 24, 2020

Back to Telescope

It's live!

Yes, Telescope is live, only its development version though, but it is live, and you can check it out here. If you have no clue about what Telescope is, go check my previous posts to learn about it. Go, I'll wait.

Ready? So here we are again, trying to add more features to Telescope, learning new things, and getting excited again!

It's becoming a very intense project, so I'll go right away to the things I've been involved with:

- GraphQL

 GraphQL is a query and manipulation language for APIs and a runtime for fulfilling queries with existing data. Adding it to Telescope wasn't easy, but it's done, and now we can take advantage of its features.


- Hashing and encoding IDs

This one was a bit trickier. It was my first time doing this so I had to get a bit familiar with crypto, a wrapper for OpenSSL cryptographic functions, but once I understood what had to be done (and with some extra help from others involved in the project), I managed to integrate it with what we had.


- Minikube and Kubernetes

Minikube and Kubernetes additions to Telescope are still WIP, but we've made some progress with them. I'm working with another contributor to run Telescope using a Kubernetes cluster, but due to our lack of experience, we're trying first with minikube. I think we're getting there, we did some testing and we hit some walls that you're supposed to hit when you're learning this stuff. If everything goes well, and our guesses are correct, I think we'll be able to get it run in a week or two.


- Deployment

This is another very interesting piece. Right now Telescope is running on port 80, but it'd be nice (and professional) to use SSL, right? Well, that's what I'm going to be doing soon. I just started researching Nginx to see how it works and how it can be used with docker-compose (which is what we using for our staging server), and once Ngingx is added to our docker-compose file, we'll try to convince Let's Encrypt that we're trustworthy.

That's pretty much what's happening right now, Telescope keeps growing at a pace that's almost hard to keep up with, but what we're getting out of it in terms of experience and knowledge is simply awesome.

Oh, and we're adding the fanciest toy in the store right now, Gatsbyjs.

Stay tuned!
Share:

Saturday, December 7, 2019

Telescope: Up and Running


Yes, Telescope is ready to start taking its first steps!

After a couple of weeks of hard work, Telescope has finally become a functional piece of software. All the elements, or I should say all the effort made by all the contributors, was finally put together and now it is possible to run telescope and see it do what it was created for.

This is really awesome, seriously. Running Telescope and being able to read posts on it is amazing. Here, take a look:





This has been so far a really awesome experience. The learning process, the collaboration, the enthusiasm from all the people involved... I'm very happy that this project exists, and I'm even happier that I'll be able to keep working on it.

As I said, the last couple of weeks were crucial to be able to get to where Telescope is right now. The components responsible for fetching feeds, process and store them in our database were connected to the front end through our web server, using 2 endpoints that I added. I had a lot of fun adding those 2 endpoints and tests for them. I had to learn about how to implement paging and adding link headers to the server's response, and a little bit of redis, which were both completely new to me, and added valuable knowledge to be used in the future. I also had the chance to get involved with the docker side of the project. 2 awesome contributors worked hard to add redis and SAML2 support to our docker workflow, and I was fortunate enough to have the chance to add some suggestions and comments here and there.

I think I have a better understanding of how redis and SAML2 can be incorporated to a project, and although I've worked with docker before, I feel more confident about using it now. That's also another part of this project that makes it so valuable. I started contributing to Telescope at the very beginning, and just by reviewing other people's work, I had the chance to learn so many new things. As an example, before telescope, my relationship with CI was almost non existent, but just by reading other people's code, their comments, and following all the steps they had to go through to add CI to Telescope, I'm ready to start adding my own contributions in this area. That right there is invaluable experience for my future in this industry.

So there it is, Telescope is really getting there. Yes, bugs will come up, adjustments will have to be made, but it is ready to see it work and do what it's meant to do. Isn't it exciting?

Good hacking and have fun!
Share:

Sunday, November 24, 2019

Telescope: Quick Update

Telescope keeps growing, and soon we'll have to start thinking about putting all the pieces together.

Not much has happened lately in terms of new features or important milestones. I started researching how to use docker on Windows to be able to add all the steps to the documentation for contributors who use Windows as their main system (my condolences to you all), and soon I'll be adding 2 new endpoints to the back-end so all the pieces responsible for processing feeds will be connected to our database. I think it'll be an interesting and challenging task, but I'm really looking forward to it.

Oh, something interesting happened. Our CI broke, and the reason was the most unexpected one. It's not that our CI hadn't broken until this week, but it was always us. We've had problems with Windows CRLF, file paths, not mocking tests, but it was always us. But this week, a new version of node came out, and for some reason it broke all our tests. We managed to fix it by making our CI use the LTS version of node, but for someone like me who barely works with web technologies, this was very unexpected.

And that's pretty much it. In a few weeks, we might have all the different pieces working together and maybe it'll be time to start testing telescope as a whole. I can't wait for it!

As usual, good hacking and have fun!

Share:

Monday, November 18, 2019

Telescope: growing faster than expected

Telescope has been alive for 2 weeks, and its development couldn't get more exciting!

I mentioned in a previous post how the project seemed to be a bit chaotic at the beginning, mostly because of the number of people involved in it and the different sets of skills and experience that each contributor has.

Well, that was last week. This week, we have a very enthusiastic group of developers exchanging ideas, offering help, being passionate about their contributions, and all that with a surprising level of organization completely unexpected (at least to me) for a group this big, and we're taking about between 50 and 60 people. As I said before, it's been fascinating to watch it grow, and I couldn't be happier to be part of it.

But there's more. At a personal level, I've been able to contribute to many parts of the project, and in may different ways! Back-end, containers, documentation, testing, CI... all that in just 2 weeks! In some cases, I added features that will enhance the capabilities of the project, in others I just made small changes or suggested someone a different approach for what they were working on. It's been a very enriching experience because not only I had to review and refresh previous knowledge to be able to offer some help, but also I had to do a significant amount of self-teaching so I could take on certain issues and complete some tasks.

So far:

Also, I reviewed other people's contributions, I collaborated with other developers to add docker to the project and write documentation about how to use it, I gave some advice about how to add tests using mock responses... and this is only the beginning! In the next 2 weeks, I'll be able to do some more work, and the best part is that even if I'm not directly involved in some part of the project, I'll be able to review or at least take a look at what others did to solve some problem and learn a lot from it. What else could I ask for?

The only downside of all this is that I got a bit too distracted and I didn't contribute to another projects, as I was hoping to. My plan was to work on a PenguinV's issue, but since I didn't, I'll try to get it done this week, and hopefully take on another PenguinV's issue next week.

As usual, good hacking and have fun!

Share: