The most important part of developing a design is to first thoroughly understand the problem that needs to be solved. The problem, itself, needs to be well-understood. (For instance....A water supply shortage of "xx" for a city is forecast.) Equally important is to factor in any major constraints that may limit the approach or type of solution that could be applied. (Potential solutions: There is a nearby river, but an endangered fish species prevents the removal of any further water from that river for drinking supply. However, there is an available river located at a considerable distance from the city that could be used. Finally, a groundwater aquifer underlying the city could potentially be used.)
After carefully studying the social, environmental and economic cost of all the potential solutions, the "best" solution is chosen. Only then does the engineer begin the detailed design process. Each engineering discipline has a set of well-understood design approaches and procedures that he/she can apply to develop the specific project design that most efficiently solves the problem. This, of course, is the idealized outcome. In the real world ideal outcomes seldom seldom materialize and that's often where past experience in a profession is critical.
Pete Sturtevant, P.E.
What a great question this is Caitlin! And, as with all great questions, the answer has many layers.
The first part is education, and mentoring. We study common sets of real world problems and patterns that solve them, whether via an algorithm or technology or combination. We combine things we learned for them in new ways to handle variants. If we're lucky (or smart) we find senior people and ask them questions. We constantly are studying to learn about new techniques or challenge our assumptions we thought we had.
The second part builds on this and gives us an experience base we can apply to new problems.
And finally, there is an element of intuition, coupled with the experience and education. I may go through several designs and test them against the problem and variants and not be happy, finding a variant that is too difficult to "work around". But, at some point, I handle most of the "use cases" (requirements) well with my design and when I come up with a challenging variant, maybe it isn't a frequent case, so I am willing to compromise.
Hope that's helpful. I don't mean to make it sound like voodoo. It's something you learn as you get more and more senior in your career.