After 8 years of working in an agency developing around 20 projects for 10 different clients, I learned a thing or two about starting new projects and accelerating the delivery process. Teaming up with new colleagues, I realized that I’ve been lucky to have so many different learning opportunities. With this article, I want to share my learnings and most important topics when starting a new project with you. I hope you can add one or two things to your “starting a new project”-toolbelt. These are not sorted in a particular order although some are connected to other points for obvious reasons.
Make your work visible
Throughout all those points, you might see a common, underlying theme: “Don’t be surprised” and “Ship faster”. These ideas are either about increasing your productivity – or rather your team’s productivity – or it’s about decreasing risks (bugs, missed deadlines, poor quality). One of the ideas that I find most important is: Have a common state of work, so everyone on the team always knows what the current progress is.
Deploy your work as soon as possible
This idea forces you to have a running application as early as possible. No one should be surprised by missing deployment pipelines, server environments, or the fact that your applications simply do not work after 3 weeks of programming. Also, this gives you the opportunity to have stakeholders or the Product Owner have a look at the application. They can experience the application and give feedback on your implementation. We’re not doing waterfall programming where no one sees the application until launch and is then absolutely surprised by the result.
The idea is not new! Eric Ries published the first version of “The Lean Startup” over 10 years ago. One of the most influential ideas was the concept of having a Minimum Viable Product (MVP), measuring and pivoting from there. The same is true for lean software development. Start with an empty frontend, or just with the login and go from there. At least, everyone knows that your login works and that the test environment is running smoothly.
“Works on my machine” does not count!
Speaking of test environments: your local machine is not a viable environment. The fact that something is running on your machine doesn’t make it run on production. No one (maybe not even you) know how exactly your local system is configured and why it is running. Be it some manual port bindings, or additional modules in your PHP configuration, even running both the frontend and backend application on the same host can hide the fact that you may run into CORS issues on production!
A first step to a more streamlined process is certainly to use some kind of common development environment (e.g., Docker containers, virtual machines if you feel nostalgic…). All configurations should be as close to the production system as possible and committed to your repositories. But even then, it’s just “close to production” and not “production”. Therefore, “but I can show you that it works on my machine” is not a sentence that should ever come over your lips.
Push your work (at least) daily
Additionally, your local environment cannot be tested by another person. For this reason, it’s necessary to push your work and deploy it to a test environment. Now let’s talk about frequency! Why is it so important to regularly push your changes to your remote repository?
I had one scenario lately. I had to help out on a project because one of the main developers got sick. Over the day, I had meetings to attend, so I tried to resolve a few issues in the evening. As I expected that this was the current state of the application, I adjusted an API call to fit the expectations of the frontend. The next morning in the standup I learned that the other developers did a pair programming session the day before and already adjusted the frontend to fit the API call. So my effort was absolutely worthless. And the only reason it happened was that the latest changes were not yet pushed.
Another reason could be that you will get sick, there’s a huge power outage, or your computer just died with all your code that you haven’t pushed yet. As I mentioned in the beginning, one of the main themes is: “Don’t be surprised”. Experiencing any of the abovementioned incidents should not have a huge impact on your project’s status. Having to rewrite big parts of your application because you did not push it regularly is not only annoying but also unprofessional. So, push your work regularly and create a WIP/Draft-Merge Request so your colleagues know what you are working on.
Programming is not Lego!
This misconception is something I experienced a lot over the past few years since everyone moved away from monolithic applications that contained both frontend and backend:
“We’re almost done. We just need to connect our frontend and backend!”
I mean sure, it’s purely logical. If you defined a proper interface how your frontend and backend are communicating with each other and everyone was 100% aligned with the definition, it should be as easy as putting two Lego pieces together and suddenly everything works. (On a side note: when you follow the aforementioned tips, you should already have a working and fully communicating front- and backend application, and – again – you won’t get surprised by misalignments between the two (or multiple!) systems).
While in theory, your architecture should be so robust that it should be as compatible as Lego, I would not risk it. Get rid of the idea that there are no impediments when connecting two systems for the first time! The longer you wait, the more complex it gets to a.) connect everything and b.) test everything. I experienced this a few years ago in another client project: the frontend developers worked on a “frontend-only” React prototype for about two or three months implementing the design requirements and adjusting it to the client’s wishes while the backend developers were busy doing other things. Then, in the mind of the project manager, it would just take a week or two to connect it to the backend application so that everything runs. The project launched about 6 months later…
Continuous Integration / Continuous Deployment
First things first: it does not matter which CI/CD tools you use. GitHub Actions, Gitlab Pipelines, Bitbucket/Bamboo, TravisCI, Circle CI, Jenkins, TeamCity – there are a lot of options! The only important thing is that you know how to use it and that you use it! Linting your code, making sure that your changes didn’t break your tests, and automating the build and deployment process is just nothing you should still do manually. This plays into both themes “Don’t be surprised” and “Ship faster” because it defines a clear process that needs to be followed and you don’t even have to do it on your own! So, there should never be a “Whoops, I forgot to run the tests before I deployed my newest changes” or “Oh no, I ran the migrations before backing up the database beforehand!” anymore. These things can and should be automated.
Pipelines
Pipelines make sure that you, as a human being, don’t forget certain steps and take care of certain tasks. Automation will give you additional time while still maintaining a level of quality that you might otherwise “not have enough time” for. Usually, your pipelines should consist of multiple stages:
- Setup
- Starting a container/runner
- Creating a database replica
- Running fixtures
- Installing assets
- etc.
- Testing
- Running static analyzers / linters
- Running unit tests
- Running functional tests
- etc.
- Build
- Building your assets (compiling, uglifying JS code, minifying CSS)
- Building your Docker images
- Tagging your build
- Deploy
- (when merged into
develop) Automatically deploying to TEST - (when merged into
main) Manually deploying to STAGE - (when merged into
main) Manually deploying to PROD
- (when merged into
This is just a rough overview but it should give you an idea of how you could structure your jobs. You could also run post-deploy jobs! This is very interesting for running regression tests! With this you can make sure everything is still working after your new release has been deployed to PROD. You could, for example, create a Postman suite for your API calls and run it automatically in your pipeline with Newman (https://moritzwachter.de/2021/please-mr-postman/).
Static Analysers
Linters are probably another quick win when you already have a base setup of jobs in your pipeline. Static code analyzers are not interpreting your code but simply check your syntax for logical, syntactical, or stylistic errors. When starting a new project, it’s best to add it from the very beginning, so you don’t have to fix certain code style issues manually afterwards. You could also make your linter fix the most obvious, non-critical issues (like adding a ; or redundant “ “ space). It’s also good practice to add it to your git pre-commit hooks. That way, your changes will be checked before leaving your computer and you might have fewer failing pipelines.
Tags
When you have CI/CD pipelines already installed, the task to add tags to your builds will probably take you 2-5 minutes. But simply tagging your builds for development/production is not the goal. The goal should be to make everyone aware of the version of your software that is currently rolled out on which system. Therefore, make your tags visible. This is the system I would normally use: {major version}.{release / minor version}.{pipeline / hotfix}
Let’s be more concrete: your major version is either 0 before launch or 1 after releasing your first product. Your release number should be manually adjusted so it fits your current sprint/release. The “patch” part – if we’re talking in “semantic versioning” terms (https://semver.org/) – should be the id of your current pipeline. Furthermore, this tag should be visible in your footer, response headers, or markup somewhere, so that everyone on your team knows exactly what has been deployed on which system by checking the pipeline ids and merge requests. Additionally, if you want to be more precise in some situations, you can tag your hotfix, release candidate, etc. manually. So, your tag could be something like 1.3.87594 for your current pipeline of the 1.3 release, or 1.3.1 if you have a specific hotfix version you want to deploy on production.
Surely, you will find a system that works best for you and your circumstances. The only important thing is: making it transparent. Make sure no one is surprised by the changes that are visible in your dev/stage/testing/production environment.
Commits, Branches, Merge Requests, and Labels…
When working together on a project, you should use the features that you have at hand. You should avoid conflicting changes by working in a separate branch for each feature. This also makes sure that your changes are only merged when they are ready. The risk of breaking someone else’s working copy after a pull should be as little as possible. With proper git messages or even conventional commits, your team will get a good understanding of what you have been doing.
Now that your branch is pushed, it’s time to create a Merge Request. Even if you working on a repository alone (for now), I still believe it has benefits working on it as if you had someone working with you or someone will join the repository later. By this, it’s still easy to understand what happened in the past and you have this “review step” you might otherwise just ignore.
When working with Merge/Pull Requests, it’s always a good idea to have a proper structure so you can filter your MRs more easily and everyone on your team knows about the state of all Merge Requests and who should be responsible for it.
Here are some examples to give you an idea of what I mean. Surely, all of these are more or less optional and should only serve the purpose to give you a bit more structure:
Review required
(This signals that someone needs to review this Merge Request before it can be merged)RTM
(”Ready to merge” – basically done, approved, waiting for pipelines to finish.)Pending assignee / Pending author
(Not quite ready yet. Someone – the assignee – needs to change/fix something before it can be reviewed again. For development speed’s sake: think about doing a non-asynchronous code review)Backend/Frontend/API/Module X/DevOps
(This is used to quickly filter which part of the application will be affected. Usually, this should be used with a very broad scope)On hold
(Waiting on some clarifications or another feature to be merged first)Needs: ...
(Maybe your MR lacks documentation, tests, DB migrations, … highly depends on your project’s needs)Release
(I like to mark my release Merge Requests fromdeveloptomainwith an additional, red label. That way, they are more easily distinguishable from other MRs in our list)
Greenfield, but…
After reading the CI/CD part, you may think: “Wow, that sounds like a lot of work! Setting up all of this will slow me down more than it’s helping me to ship code faster!”. Now, here’s the thing: even if you start a project from scratch, you should not start from scratch! This is especially true when you’re working on a time-sensitive prototype. If you have a timespan of one year ahead of you, take your time and “sharpen your tools” (e.g. CI/CD) before you start coding. But if you’re just doing a small project, you shouldn’t reinvent the wheel. Instead, start with a code base, starter project or bootstrapper to get started quickly. When time is scarce, you should not try to rebuild your whole scaffolding.
Things that could already be pre-configured when starting with your project:
- Docker images to use
- Docker Compose orchestration of your containers
- Helm Charts for using Kubernetes
- CI/CD Jobs & Pipelines
- Makefile with your most common tasks
- Deployment scripts
- npm / composer / Maven / Gradle packages
- a README explaining how to run your base setup
- SSL certificates to use HTTPS locally
- and more… (you probably have some ideas in mind)
As long as you’re not chasing after the next fancy Javascript framework with every new project that you start, this means a one-time, quality effort for a plethora of new projects to start.
Write tests
Now, writing tests is always a fun topic! From my experience in interviews, a lot of developers out there see the benefits of having tests and generally like the idea of writing tests but it’s still not as widespread as I would have expected to have a proper test suite. As we are talking about starting new projects and, as mentioned before, these projects might have some time pressure, the common argument is: We don’t have time for tests.
Let me put this as bluntly as possible: Tests will save you time because you don’t have to manually test things again, and again, and again. From the moment a test runs some assertions without you doing anything, it saves you time! If you’re still not convinced that it’s worth the invested time, it might be because you’re not very good at writing tests. That won’t help you if you’re looking for advice on starting a project right now, but please keep that in mind and invest in becoming better at writing tests!
Writing tests for new projects might be different than writing tests for an existing project. There might be other priorities and what needs to be tested and what doesn’t. Also, I would highly recommend using TDD or BDD to implement some important business logic and behavior. It not only forces you to have your accepted behavior tested, but it will also automatically act as a regression test and give your a great scaffolding for writing other features that might break your code. Consider writing Smoketests that are easy to write. In one of the later chapters, I will talk about APIs. It should be fairly easy to create a Postman suite from your OpenAPI specification.
With this, you already have ready-to-use smoke tests for your API that can be run on every pipeline! But keep in mind to have a healthy mix of integration/regression and unit tests. All unit tests without any functional tests might break your system (although your tests were green). All integration tests without unit tests might be working for your current implementation but hide a lot of technical debt (e.g., writing services that are not properly testable). On top of that, it might take a while as integration tests are – by nature – slower than unit tests.
Mission-critical functionality
Your mission-critical features will be tested the most thoroughly. So, it’s just logical to automate these test cases as well as possible to save time and not be surprised! I remember my very first project when starting as a Backend Developer. We were building some health/fitness websites with a paid, logged-in area where you could track your fitness progress, get meal plans, and watch videos.
What did our project manager do after every deployment? (Keep in mind, this was like 2013/14…) She opened the website, paid for the product with her personal credit card, entered some mock data in her profile, clicked on a video to check if it was still playing, and asked our client to refund the purchase to her. This sounds like an awful waste of time if you ask me. While I absolutely appreciate her dedication to making sure everything works for our client’s customers, I still feel this could have been automated!
So what are your mission-critical features? How can you automate them? Sorry for repeating myself: Don’t be surprised! Fixing bugs after they have been deployed to production causes a lot of noise and can be mostly avoided with proper (automatic but also manual) testing.
Risk Assessment
What could go wrong?
A few years ago, I read “Rapid Development” by Steve McConnell. This book is from 1996! And it made me absolutely speechless how many things we’re still doing wrong/badly over 25 years later! It covers a lot of very important topics when planning and implementing a software project. But one of the topics that I found most important was “Risk Management”. As we don’t want to be surprised, we should ask ourselves: “What are the worst things that could happen to our project? How likely are they to happen? What can we do to prevent them?”. Identifying risks early, prioritizing them, and thinking about ways how we can avoid them happening before they happen, is something I see far too less in software development projects all around.
Be prepared for the worst
Projects are never risk-free. There are financial constraints, time constraints, colleagues might become parents or sick for several weeks, etc., etc. As a result, you should identify your TOP 5-10 worst-case scenarios. Write them down. And then list action items you can a.) do beforehand to lower the risk/impact and b.) what you will do if the scenario has happened. With this, you might not be prepared for everything but at least you should be prepared for the risks with the highest impact on your timeline/budget.
Collaborate
Surely, most of the things I wrote above are already things to consider when you work together with other people. So, if you skipped the section(s) above, it might be worth jumping back. This part will be more about topics that are outside of your IDE. Nevertheless, they are as important as the topics mentioned above! From my experience, most projects don’t struggle because of missing technical expertise or lacking morale but rather because of inefficient communication and missing technical guidance/leadership.
When you put 3 developers on a project, someone has to take the role of a ‘Lead Developer’. I don’t mean “solving the most complex (and most exciting) issues”! I mean coordinating the team, identifying risks, removing blockers, and making sure that the project is run properly even though this is not related to coding. So, if you’re not tracking issues, if people are not talking to each other, if deployments are unstable, or if there are any other red flags appearing, this is your responsibility, too!
Write a README & CONTRIBUTING.md
Being prepared also means that it might happen that someone else needs to assist on a project because one of the main developers called in sick for a week or longer and you did not account for this in your time planning. In this case, it’s crucial that this person can get as much information as possible without asking someone. You should write down all the things mandatory to properly run and contribute to your project. And I’m not talking about in-depth documentation! In most cases, you’re probably not starting a rocket science project, so someone from the same domain or discipline should be able to get into it easily. This is important for both “Don’t be surprised (as a new developer joining the project)” and “Ship faster (as a new developer joining the project)”. No one wants to help out on a project and waste hours figuring out why OAuth is not working because nobody told them to add the JWT keys (or any other not-so-funny surprise). Therefore, document all necessary steps to make your project run. To speed things up, you might consider writing a Makefile or something similar to bundle your most-used tasks together.
The same is true for a CONTRIBUTING.md. In order to get uniform code style, commit messages, branch naming, tagging, etc., it’s important to summarize the most important rules in one file. Ideally, you can start with a file you already used in another project and change it on the way if necessary. There shouldn’t be too many changes necessary. If you never used a CONTRIBUTING.md, I highly recommend it! Find more information about what to put in that file here: https://mozillascience.github.io/working-open-workshop/contributing/.
Use an issue tracker (of some sort)
The amount of different tools that you can use to do ticket management these days is huge! So let me start with my first tip: If you have a tight timeline, don’t start fooling around with a new fancy project management tool! Stick to what you know best. And also: it does not have to be JIRA! Obviously, the choice of your tool might highly depend on the client, the company you work for, or the size of the project. Gitlab Issues might be a good idea for smaller development projects but it wouldn’t be my first choice for a team of 15 people. The same goes for Excel. 😉
Nevertheless, even Excel is better than not having written down your open tasks and issues! If you know, you will work on a long-term, huge project, this might be insane. But if you’re in crunch time and already behind schedule, Excel can be a good way to at least get an overview of what’s missing, so everyone sees either a table of all “reds” (i.e., open features/bugs), some progress, or all “greens” (i.e., finished tasks). (This message is also for Project Managers and Business Analysts): If you don’t have time, please don’t start creating fancy dashboards, a nice ticket workflow, and three meetings to explain to others what you have done. Keep it simple (and quick to use)!
To do a little sanity check on your issue tracker usage, just answer these two questions:
- Is it helping us to not be surprised (e.g., by forgotten issues, or missing features)?
- Is it helping us to ship faster or is it slowing down the development/delivery process more than it’s helping?
Use Mock APIs (and define your API together)
When working with distributed systems, it happens just so often that one (or both) of these scenarios happen:
- “I’m blocked because I have to wait for the API call to be finalized. There are fields missing I absolutely need!”
- “The other part implemented it wrong! Now, I have to adjust the API that was already finished!”
This should really not happen when you define an API standard together in the beginning. You could start be writing down your endpoints in a Swagger file: https://swagger.io/specification/. Ideally, you use a changelog and versioning to keep track of API changes. Then import this OpenAPI specification into Postman, and create a Mock Server out of these calls! No coding, (almost) no manual work. If something changes, re-import the specification, adapt the mock server, done. I cannot think of any reason why you should ever write your own mock API (which takes additional time) instead of using services and solutions that are ready to use.
Work in iterations
Most of us nowadays probably heard “We’re working with SCRUM” much more often than it’s actually true. So, this should not be a surprise to you: work in iterations of X weeks, review your progress, set the next milestone, and don’t be surprised by the deadline all of a sudden. Whether you use a one-week, a two-week, or even a four-week sprint completely depends on your project’s timeline, the size of the team, the risks of having changes/feature requests from the client, and how you want to work as a team. In the end, the process has to fit you as a team. If you spend more time in meetings than delivering there’s clearly an issue with how or how many ceremonies are held.
Sync on a daily basis
This might be another no-brainer for those who are accustomed to agile/Scrum practices. But I need to stress that among all ceremonies, having a properly held standup is probably the most important meeting. It should give everyone an understanding of what your current status is. On top of that, blockers can be identified quickly and should be solved by the team with urgency so no one is blocked from their work. It’s also a great way to get a feeling for the team, see each other face-to-face for a few minutes, or even socialize!
There are numerous ways how to do a daily and it (again) highly depends on your way of working as a team. You could use an app (https://standuply.com/), you could do it asynchronously (https://www.range.co/blog/asynchronous-daily-standups), or do it in person. It should be short, and focused, but I would recommend keeping it human. We are no machines, let’s have some fun! Use Ice Breakers, tell a bad joke, or have some weird rituals that bond you together. Maybe a team anthem that you play every morning? In one of my former teams, we introduced a ritual that our Project Manager would start with “Guten Morgen” and the whole team answered as a (very slow) choir “Guuuuten Mooooooooorgen” (imagine like back in the days at school). We did this in person, with a team of around 10 people in an open office. The looks and laughs from everyone around were priceless 😅. In summary: do something that makes you feel good! Use it as an anchor to start a great and productive day!
Escalate early
This is probably one of my biggest learnings from the last year. If you don’t feel comfortable if you have a gut feeling of “Oh, this is not looking good”: talk about it. Escalate it! Just to be clear, I’m not saying “Complain about the rough timeline every day!”. But sometimes other people on your team are not aware of the risks and potential impact a certain issue might have. The worst thing that could happen is that everyone raises their concerns afterwards. “Afterwards” means it’s too late! So raise your concerns early and raise your concerns again if you feel ignored. Other people don’t have to agree with your concerns but at least they need to hear it!
Don’t be surprised, and ship faster!
Wow, a whole article summarized in one headline! But probably, it’s an even more complete representation of all the things to keep in mind when starting a new project. I’m still on a journey, so see this article as a current snapshot of my learnings. Hopefully, they included something to learn for you, too!
Now, what about you? What are your most important pricinples to keep in mind when starting a new project? Did I miss something important?
