Showing posts with label cdot. Show all posts
Showing posts with label cdot. Show all posts

Sunday, March 1, 2020

Telescope: Elasticsearch and Automatic Deployment

For the 0.7 release, I've been working on a couple of issues: adding text indexing and automatic deployment.


Text indexing
Adding text indexing to Telescope was interesting. After checking out some options, Elasticsearch (a distributed, RESTful search and analytics engine) was the tool that we decided to use to implement text indexing. I added a new container running Elasticsearch to our workflow that will be waiting for requests to index blogposts or search for already indexed blogposts that contain a passed string.

Surprisingly, the main problem I had to deal with was not functionality or interactivity with Elasticsearch, but mocking it. Elasticsearch works great with Telescope, but a fourth of our tests fail due connectivity problems (Elasticsearch is down when I run tests, so the client complains). I did a bit of research to find out about use an Elasticsearch client in mock mode to mimic what we do with redis, but from what I've seen it's not as easy as I thought it'd be and it might involve adding some Java to the mix, which is something that we don't really want for Telescope.

I'm still working on a solution (I'll have a PR ready soon), but I think I'll wrap the Elasticsearch pieces in our code in conditionals that will only run when an environment variable is set to a specific value.


Automatic deployment
For 0.7, we agreed to add simple auto deployment. The ultimate goal is to have blue/green deployment, but since that'll probably involve deep changes to our current docker settings I think it's a good idea to take this first step and then try to build on it in future releases.

To implement automatic deployment, I added a small server that waits for requests from GitHub to shutdown, reclone, and redeploy Telescope whenever a pull request is merged to master. For this, we're using GitHub webhooks (Calvin I worked on this part together).

I got this piece to work on my machine, but I still have a small issue with our .env file. Hopefully I'll be able to fix it soon and add this to our staging server soon.


Other issues
I also did a bit of work to have persistent storage with redis and adding some more filters and nested queries to our GraphQL. I think these 2 will very likely be included in the 0.8 release.
Share:

Saturday, February 1, 2020

Telescope: Nginx and Let's encrypt

So, Telescope is getting closer to become the successor for PlanetCDOT. If you still don't know what Telescope is (shame on you), check this out.

Nginx & Let's Encrypt
I spent this week researching nginx to add a reverse proxy to our staging box so we could add some cool features. Also, I worked with another contributor (@miggs125) to add Let's Encrypt so we can use SSL.

After dealing with some configuration issues, we managed to get it to work, and once we merge the necessary changes, we'll have Telescope using SSL. I think I've probably said it too many times, but once again thanks to Telescope, I had the change to learn new things: I got to learn how to setup a reverse proxy and how to add SSL to a site.

Feed endpoints
As I mentioned in previous posts, Telescope has a REST API and GraphQL. The endpoints to serve posts were implemented some time ago, but we were still lacking a way to serve feeds to our frontend. Since I collaborated adding the endpoints for posts, I thought it'd be a good idea to work on the same issue but for feeds. I added endpoints to request feeds and one feed, along with some tests for those endpoints.

GraphQL filters
I also did some work for GraphQL. Another contributor (@c3ho) is working on adding filters to our Apollo server, so our queries will be more flexible and efficient. It's a very tricky task, so I offered to help since I worked on the initial addition of Apollo to Telescope. We've made some progress and, to be honest, if he manages to include all the functionalities we want (taking advantage of some Apollo's features), the improvement will be significant. Seriously, really interesting stuff.


Share:

Friday, January 24, 2020

Back to Telescope

It's live!

Yes, Telescope is live, only its development version though, but it is live, and you can check it out here. If you have no clue about what Telescope is, go check my previous posts to learn about it. Go, I'll wait.

Ready? So here we are again, trying to add more features to Telescope, learning new things, and getting excited again!

It's becoming a very intense project, so I'll go right away to the things I've been involved with:

- GraphQL

 GraphQL is a query and manipulation language for APIs and a runtime for fulfilling queries with existing data. Adding it to Telescope wasn't easy, but it's done, and now we can take advantage of its features.


- Hashing and encoding IDs

This one was a bit trickier. It was my first time doing this so I had to get a bit familiar with crypto, a wrapper for OpenSSL cryptographic functions, but once I understood what had to be done (and with some extra help from others involved in the project), I managed to integrate it with what we had.


- Minikube and Kubernetes

Minikube and Kubernetes additions to Telescope are still WIP, but we've made some progress with them. I'm working with another contributor to run Telescope using a Kubernetes cluster, but due to our lack of experience, we're trying first with minikube. I think we're getting there, we did some testing and we hit some walls that you're supposed to hit when you're learning this stuff. If everything goes well, and our guesses are correct, I think we'll be able to get it run in a week or two.


- Deployment

This is another very interesting piece. Right now Telescope is running on port 80, but it'd be nice (and professional) to use SSL, right? Well, that's what I'm going to be doing soon. I just started researching Nginx to see how it works and how it can be used with docker-compose (which is what we using for our staging server), and once Ngingx is added to our docker-compose file, we'll try to convince Let's Encrypt that we're trustworthy.

That's pretty much what's happening right now, Telescope keeps growing at a pace that's almost hard to keep up with, but what we're getting out of it in terms of experience and knowledge is simply awesome.

Oh, and we're adding the fanciest toy in the store right now, Gatsbyjs.

Stay tuned!
Share:

Saturday, December 7, 2019

Telescope: Up and Running


Yes, Telescope is ready to start taking its first steps!

After a couple of weeks of hard work, Telescope has finally become a functional piece of software. All the elements, or I should say all the effort made by all the contributors, was finally put together and now it is possible to run telescope and see it do what it was created for.

This is really awesome, seriously. Running Telescope and being able to read posts on it is amazing. Here, take a look:





This has been so far a really awesome experience. The learning process, the collaboration, the enthusiasm from all the people involved... I'm very happy that this project exists, and I'm even happier that I'll be able to keep working on it.

As I said, the last couple of weeks were crucial to be able to get to where Telescope is right now. The components responsible for fetching feeds, process and store them in our database were connected to the front end through our web server, using 2 endpoints that I added. I had a lot of fun adding those 2 endpoints and tests for them. I had to learn about how to implement paging and adding link headers to the server's response, and a little bit of redis, which were both completely new to me, and added valuable knowledge to be used in the future. I also had the chance to get involved with the docker side of the project. 2 awesome contributors worked hard to add redis and SAML2 support to our docker workflow, and I was fortunate enough to have the chance to add some suggestions and comments here and there.

I think I have a better understanding of how redis and SAML2 can be incorporated to a project, and although I've worked with docker before, I feel more confident about using it now. That's also another part of this project that makes it so valuable. I started contributing to Telescope at the very beginning, and just by reviewing other people's work, I had the chance to learn so many new things. As an example, before telescope, my relationship with CI was almost non existent, but just by reading other people's code, their comments, and following all the steps they had to go through to add CI to Telescope, I'm ready to start adding my own contributions in this area. That right there is invaluable experience for my future in this industry.

So there it is, Telescope is really getting there. Yes, bugs will come up, adjustments will have to be made, but it is ready to see it work and do what it's meant to do. Isn't it exciting?

Good hacking and have fun!
Share:

Monday, November 18, 2019

Telescope: growing faster than expected

Telescope has been alive for 2 weeks, and its development couldn't get more exciting!

I mentioned in a previous post how the project seemed to be a bit chaotic at the beginning, mostly because of the number of people involved in it and the different sets of skills and experience that each contributor has.

Well, that was last week. This week, we have a very enthusiastic group of developers exchanging ideas, offering help, being passionate about their contributions, and all that with a surprising level of organization completely unexpected (at least to me) for a group this big, and we're taking about between 50 and 60 people. As I said before, it's been fascinating to watch it grow, and I couldn't be happier to be part of it.

But there's more. At a personal level, I've been able to contribute to many parts of the project, and in may different ways! Back-end, containers, documentation, testing, CI... all that in just 2 weeks! In some cases, I added features that will enhance the capabilities of the project, in others I just made small changes or suggested someone a different approach for what they were working on. It's been a very enriching experience because not only I had to review and refresh previous knowledge to be able to offer some help, but also I had to do a significant amount of self-teaching so I could take on certain issues and complete some tasks.

So far:

Also, I reviewed other people's contributions, I collaborated with other developers to add docker to the project and write documentation about how to use it, I gave some advice about how to add tests using mock responses... and this is only the beginning! In the next 2 weeks, I'll be able to do some more work, and the best part is that even if I'm not directly involved in some part of the project, I'll be able to review or at least take a look at what others did to solve some problem and learn a lot from it. What else could I ask for?

The only downside of all this is that I got a bit too distracted and I didn't contribute to another projects, as I was hoping to. My plan was to work on a PenguinV's issue, but since I didn't, I'll try to get it done this week, and hopefully take on another PenguinV's issue next week.

As usual, good hacking and have fun!

Share:

Sunday, October 27, 2019

Hacktoberfest: Documentation

We're in the last week of Hacktoberfest!



So far, participating in Hacktoberfest has been great. I've been able to contribute to very different projects and work towards the goals I had set for myself.

My work on Dashboard allowed me to improve my web skills (not as much as I'd have liked, but still) and collaborate in a project that has been around for some time. It's not a big project, but it's a very interesting one because of all the different components that it has and how useful it is for the people involved in it.

PenguinV was my chance to collaborate in a project that I really like, and also a great opportunity to face a real, scary challenge. Having to contribute using OpenCL, something I wasn't familiar with at all before, was one of those things than can easily make you think that you shouldn't be in this field.

For my last contribution, I wanted to work on PenguinV again, but since we're at the end of the month and the PenguinV community hasn't responded (yet) when I told them that I wanted to work on some issues, I decided to look around to see if I could find some other interesting projects.

And I found ImCompressor. ImCompressor is an image processor that works with lossy and lossless compression. What I found interesting about this project is the fact that it's a very young one. A small group of developers started it a month ago. All my other contributions during Hacktoberfest were related to bigger projects that have been around for a while, so I thought it could be interesting to contribute to a young project and maybe become a core part of it. 

When I found the project, they were getting ready to release version 0.8, so since there wasn't much that I could offer (remember, it's a very young project), I mentioned to the team that my first language is Spanish and that I could add a Spanish translation. They welcomed the idea, and I thought it was a good way to introduce myself, getting familiar with the project, and keep an eye on new issues popping up so I could work on them and become a main contributor.

Oh, before I forget, when I started the translation I realized the ImCompressor team uses Poedit, a cross-platform, open source, translation editor. Another interesting project, I also want to keep an eye on it and maybe see if I can contribute to it in the future.

I like ImCompressor. I've had issues with lossless JPEGS in the past, and I think this project could help many people. I hope it won't turn out to be one of those many projects that dies a few months after being born.

Well, and that's all for this month. As I said in another post, I really want Hacktoberfest to become a yearly tradition, but not the only excuse to keep collaborating with open source projects.

As usual, good hacking and have fun!

Share:

Saturday, October 5, 2019

Hacktoberfest: good first issue

It's Hacktoberfest time!

As I mentioned in my previous post, I'm very excited about Hacktoberfest, not only because I'll be able to contribute to some cool open source project, but also because I'll have the chance to improve  and test my skills, and maybe try to take on some scary project that will force me to learn and get familiar with new technologies.

My intention was to work on some issues that I found interesting and relevant for the type of skills that I'd like to develop, and also getting familiar with some interesting projects.

Unfortunately, things rarely go the way you want in software development, so I had to come up with a plan B for my very first Hacktoberfest issue.

At first, I wanted to work on an issue in the home-assistant project, but I had some problems setting up the development environment, and since they haven't replied yet, I thought I could work on something else while I wait for their advice.

Since I was running out of time, and all the other issues that I wanted to work on require more than a couple of days of coding, I turned to a project that I've worked on before, knowing that I could find some interesting issues that I could fix within the available time that I had.

This project is Dashboard, maintained by CDOT's RAs.

Dashboard project (running in debug mode with fake data)

Dashboard displays information about the projects that are currently in development at CDOT, along with some infrastructure information and scheduled events. It populates some of the panels with data fetched from Github using Github's API.
For the PR Reminder panel (WIP) and the Help Wanted panel (coming soon), it was necessary to filter some of the data fetched from Github's API so the only issues and pull requests used to populate those 2 panels would be the ones with the appropriate labels. The issue that I worked on specifically requested filtering out issues that didn't have the Help Wanted label and also making sure that all the used issues included a priority label so they could be appropriately sorted.

It was a quick fix, but a very important one since the Help Wanted panel will be up soon (that's another issue I'd like to work on for Hacktoberfest), and all the RAs at CDOT will be able to have updated and sorted information about issues that might need help (hence the Help Wanted label) and/or pull requests that need review.

Hopefully, I'll be able to work on home-assistant's issue next week, but if I can't, I might take on a penguinV's issue that will help me get familiar with their code base and maybe I'll be able to contribute more in the future.

So, good hacking and have fun!

Share:

Friday, September 27, 2019

Hacktoberfest!


Hacktoberfest is around the corner, and as a software developer interested in open source, it’s a very exciting time!

It’s my first time participating in Hacktoberfest, I have no idea about how it’s going to be like or the type of work that I’ll be doing. Whatever happens, I have 2 clear goals that I’ll try to accomplish: improving my web development skills and facing a scary challenge.

I’ve been a developer for some time now, but because of the circumstances of the work I had to do, I didn’t have a chance to develop my web skills as much as I would’ve like to. Since the web is the future, I think having a solid knowledge of web technologies is crucial to be part of the software development community, and collaborating in a project heavily web-based will help me acquire the necessary knowledge to start feeling comfortable with web technologies. Also, I’ve been lucky (or “lucky”) so far because all the projects that I’ve been involved with were accessible in one way or another. With accessible, I mean that I was able to adapt to them, learning and developing the necessary skills to be able to contribute to them. Hacktoberfest gives me the chance to try to find something potentially scary that will require the best of my abilities to succeed. I don’t know if I’ll be able to overcome a real scary challenge, but I think facing one is something that every developer should go through every now and then.

These are the issues that I’d like to address:

There’s a project started and maintained by CDOT RA’s called Dahsboard that has been in development for some time, and it has a fair amount of issues related to web technologies that would definitely help me improve my react skills (I’ve never done anything in react before).
From that project, I want to take on these 2 issues: Pull Request reminder and Coding Style Standards.

Another interesting project that I’d like to work on is PenguinV. As their creators describe it, "PenguinV is a simple and fast C++ image processing library with focus on heterogeneous systems." The issue I’d like to work on is not particularly exciting, but I think it’s going to help me get familiar with the code and eventually start trying to take on more relevant issues and really contribute to the project.

Wish me luck!
Share: