Codebase size

Independant of what the codebase does it incorporates maintenance-efforts: On big codebases it takes longer to implement changes, more code needs to be read/comprehended. Codebase size also correlates with longer builds, deployment processes and startup times, which means a latency-increase of feedback loops. Encapsulation/Modularizaton can be applied, but it only weakens the impact: You won’t be able to reduce efforts of a 100M codebase compared to a 10K one by just having good code-quality.

Codebase quality

Decent code-quality is important for implementing changes quickly (readable code, separation of concerns, DRY etc.). Good quality also includes test-coverage so regression-bugs are better caught and there is a safety-net during refactorings. Though business often neglects less visibile code-quality, it is essential to reduce lead-time. In severe cases you’re at a dead end: code is such in a bad shape, that it is impossible to evolve (changes are too side-effect-risky and/or too expensive).

Tools/technology diversity

Itself diversity of tools (programming-languages, frameworks, hardware etc.) is good because you can choose the right one to solve your specific problem. But it also increases risk: single-person-know-how, legacy-tech, learning-curves, beta/buggy-stability, upgrade/patching-efforts, transitive dependencies (see .dll, .jar nightmares).

Integration points

Though distributed systems are necessary (scaleability, reliability, modularization, partner-integrations) they are more fragile: Monitoring and deployment efforts increase and security breaches are more likely. Also tracing, debugging and testing efforts are higher.

Organization Size

One of the biggest factors is the size of your organization because efforts increase squarely (see also Mythical Man Month). Bigger organizations try to fight this by introducing hierachies and heavy-weight processes, but this structure can have negative impact on “short/quick” decisions and slows down speed. In some scenarios political-games emerge, which block progress considerably.

External partners

External-partner-integrations (most likely over APIs, batch processing) need more investment: Different release cycles need to be taken into account, compatibility must be offered and dedicated monitoring needs to be setup. Also communication is less direct (fixed calls, meetings, travelling).

User-Base + Traffic

A bigger User-Base means you have to spend more effort on support. Because of the mass statistically more edge-cases come up and need to be handled by software. Also scalability requirements need to be implemented by more sophisticated production environments. Popular systems also attract criminal activity, which you need to respond with higher security-investments, which again make the system less usable. Big applications also produce more data, which needs to be maintained (compatibility, migrations, analyzation).

Being Complexity aware

Above factors cannot be reduced to Zero (you will always meet an Inherent Complexity), but you should fight back:

• Prioritize, prioritize, prioritize: Featuritis has bad impact on usability and codebase-size. Implement important features only. Remove unneeded features and cleanup code.
• Consolidate technologies: Don’t introduce a new technology just because it was praised in the last magazine. Evaluate it and maybe use it privately first.
• Take Fowler’s advice serious: “Don’t distribute, if you don’t have to”
• Don’t always increase team-size, rather have a small team with A-players.
• Quantify: Use metrics for business success, code-quality and production stability. Especially for bigger systems gut-feeling isn’t enough.
• …

The presented sections go along with my favorite test-smells:

1. Brittle tests: Though functionality hasn’t been changed the test fails. Test should show green but in fact shows red (false positive).
2. Inefficient tests: The effort of writing automated tests doesn’t pay out at all. The benefit/cost ratio (short + long term) is extremely low.

Unit-Test little scripts/tools

There is often no sense to write unit-tests for little scripts or tools, which are one or two-liners. The script content is already so “declaritive”, short and compact that the code is too simple to break. Further more often stubbing or mocking the dependencies is tough (e.g. writing to stdout/file, shutdown machine, doing an HTTP call). You can end up writing a external system emulator which is overkill in this situation. Surely testing is important but for that I go the manual way (executing script, and smoke-test sanity check the outcome).

Unit-Test high level orchestration services

Orchestration services have many dependencies and chain-call lower services. The effort of writing such unit-tests is very high: Stubbing/Mocking all these outgoing dependencies is tough, test setup logic can get very complex and make your test-code hard to read and understand. Further more these tests tend to be very brittle, e.g. minor refactoring changes to production code will break them. Main reason is that inside test-code you have to put a lot of implementation detail knowledge to make stubbing/mocking work. You can argue having many fan-out/outgoing dependencies is a bad smell and you should refactor from start on. This is true in some cases but higher order service often have the nature to orchestrate lower ones, so refactoring won’t change/simplify much and make design even more complicated. In the end for such high level services I much prefer to let test-cover them by automated or non-automated acceptance tests.

Test-first during unclear Macro-Design

When implementing a feature or something from scratch often the macro-design is blurry, I like to call this “diving-in”. For diving-in development or quick prototyping you get a feeling which design fits or not. During this phase class structures/interactions change a lot, sometimes even big chunks of code are thrown away and you restart again. Such wide code changes/deletions often will break your tests and you have to adapt or even delete them. In these situations test-first approach doesn’t work for me, writing test-code even distracts me and slows me down. Yes, unit-tests and test-first approach can and should guide your design but I experienced this counts more when the bigger design decisions have been settled.

100% Code-Coverage

I can’t overstate this: Code-Coverage != Test-Coverage. The Code-Coverage of unit-tests is a nice metric to see untested spots, but it is by far not enough. It just tells you that the code has been executed and simply misses the assert part of your test. Without proper asserts, which check the side-effect of your production code and expected behaviour, the test gives zero value. You can reach 100% code-coverage without having tested anything at all. In the end this wrong feeling of security is much worse as having no test at all! Further more 100% code-coverage is inefficient because you will test a lot of code, which is “too simple to break” (e.g. getters/setters, simple constructors + factory-methods).

Summary

Above points shouldn’t give you the impression that I do speak against automated unit-tests, I think they are great: They can guide you to write incremental changes and help you to focus, you get affinity for green colors ;), they are cheap to execute and regression testing gives you security of not breaking things and more courage to refactor. Still going with the attitude that you have to go for 100% code-coverage and to test every code snippet will kill testing culture and end up in Red-Green color blindness.

Overview of differences

Because there still is some confusion about dynamic vs. static typing here a short overview of my perception:

1. Dynamic typed programming: The types or variables are bound during runtime. This happens if you use languages which have dynamic typing in their core design (PHP, Ruby, Python). Be aware that you can still do dynamic typing with languages focusing more on static typing (like C# or Java): Simply have a look at the reflection APIs… Further more type saftey is deferred to runtime, if a programmer introduces typing errors they will pop up during runtime (e.g. ClassCastException, MethodNotFoundException, TypeError).
2. Static typed programming: The types are resolved/bound during compile time. For this check to be possible your code needs to have typing information (e.g. when defining a variable or method signature). If you have obvious type errors in your code the compilation result will catch it and inform you.

Pros dynamic typing

If designed correctly into the language, dynamic typing offers a lot of advantages:

• Code is much more compact. Small tools and scripts are a charm to write and some nice syntactic sugar constructs are possible. Programmers need less time to understand and extend code. Further more the proportion of functionality vs. lines-of-code is greater, you need to produce less code to solve a problem. Smaller codebases are a plus for maintenance.
• Sometimes dynamic typing is a necessity of solving a problem (e.g. the calculated return value is dependant on the parameter’s type, which can differ for every caller).
• Development feedback cycle tends to be shorter. Most languages offering dynamic typing are intepreted and a simple ‘Save file’ will do. In most cases you instantly will see the effect. For bigger server applications less restarts of the server are necessary.

Pros static typing: Maintainability

Above I mentioned compactness of code as a maintenance plus for dynamic typing. Still, on the same topic static typing scores significantly.

Analyzation of codebases

Typing information bloats code but also serves documentation. In many cases explicit typing helps a lot in reasoning the code’s semantics. Also the IDE can help to easily jump throughout the codebase. It doesn’t have to reason or guess but can directly guide you to the target code. Explicit typing information also does help tools like Structure101 or JDepend to automatically show you relationships, which can point to design or layering flaws.

Feedback of errors

You get faster feedback when you did a typing related coding-error, e.g. wrongly related a function/method or did a typo. The compliation phase does this for granted. Some argue that you anyway should do cover anything by automated tests and therefore can instantly catch typing related programming errors. This theoretically is true but in practice doesn’t hold, I’ve never seen convincing test suites which execute every code-path instantly uncovering such errors. Often you get such errors later by accident, in testing phase or worse during production. Compile time type safety is nothing to be neglected…

Safe refactorings/tooling

It is the “nature” of dynamic typing that you can’t find out typing just looking at the structure of the code. The opposite is true for static typing, therefore you can create tools for safe and automated refactorings (e.g. Rename method/function, Add/Remove parameters, general restructuring of classes/modules). If many refactorings wouldn’t be automated I would lose a lot(!) of time. Being able to refactor codebases in an efficient sensible time is a must have for me. The greater the codebase the greater the cost for unsupported automated/unsafe refactorings. Some people again argue that 100% test-coverage safety-net would suffice, for my point of view see above ;)

I purposely discarded the old performance comparison. It states that compiled binary-code is faster as interpreted code. It may have been true in older times, but nowadays, also with all the JIT compiling features, I haven’t seen convincing benchmarks that static typed languages binaries are better in performance in general. This doesn’t imply that you won’t hit scaling problems which are indeed bound to a language (see also podcast about facebook getting problems with PHP at some point)!

Summary

Dynamic typed languages are great but for bigger codebases I still would go for languages like Java or C# (haven’t tried out yet, but Scala seems to be a new alternative here). For very specific modules solving highly generic/dynamic problems you still can code in the “right” language. Modern runtime environments (like JVM or CLR) give you possibility to deploy common-libraries/APIs, which are written in dynamic typed language.

If you have case studies or other resources investigating dynamic vs. static typing and overall productivity or providing different opinions please comment.

Note: For simplicty throughout the post I will use java terms (class, method, package). Of course this can be mapped to other languages constructs like functions, file-includes or plain-script statements.

Advantage of more fine grained modularization:

• Maintaining less artifacts (e.g. files, packages, libraries) makes the build/deployment-lifecyle easier.
• For simple features code browsing gets easier (“You see all stuff with one eye-sight”)

The drawback is that the fine grained approach doesn’t scale: The bigger the codebase gets the more difficult it is to “see” any modularization or separatation of concerns. The coarse grained modularization gives you advantage here:

• Bigger systems are easier to comprehend if the “split” is done on package or library level.
• Refactorings are getting easier because inside the modules the direct numbers of dependencies get less (submodules only import dependencies they need).
• Unit-test setup gets easier (for reason see Refactoring)

Where to start?

The question is at which modularization level you should start. There are two major antipatterns, either developers sticked on too fine level (e.g. 6000 LOC inside one single file) or they started on a too coarse level (e.g. each of the many packages only contains one or two classes). The too coarse pattern often occurs if you overengineer solution.

Code from scratch

To avoid the too fine/coarse pitfall I follow the Inside-Out modularization approach:

1. Start to edit a single file. Implement the highest priority requirements of feature inside the most fine grained “module” your language can you offer (e.g. class-method).
2. When code inside a method gets bigger and you lose overview try to cluster statements (e.g. by speparating them with line-breaks).
3. When code statement clusters get too many and you see duplications, extract these section to a new method.
4. When there are too many methods and the lines of code inside the single file are very high, cluster your methods by problem domain and extract them to a new class in the same package.
5. When there are too many classes inside on package, either create a subpackage or sibling-package which fit to the problem domain (separation of concerns).
6. When a package hierachy/tree gets too deep and wide, create a new package hierachy (e.g. com.foo becomes com.foo.system1 and com.foo.system2)
7. When there are too many package hierachies inside one library (like .jar), create another library-project (e.g. Maven project/module).

Integrate changes in existing code

Above is a more or less complete list when starting with code from scratch. But how does it apply to existing code and integrating changes? The main principle is the same but you would start your Inside-Out modularization on a different level. As Example: If you have to add code inside a class and see it you feature-adding would result in too many methods you start off with step number four (extracting class).

At which step to level up?

It is always the question, when to level up from a fine to a more coarse grained modularization. It is very difficult to have a thumb of a rule, because this highly matters on code-style taste, on density of ‘if/else’ logic and also on the problem-domain you are trying to solve. A very good test is either ask colleagues for review whether the modularization is intuitive or take another look the next day or a week after to get a fresh view.

The following codebase size and LOC metric scope is not fine grained on function or class level but for complete codebase or at least on subcomponent level.

Bad (anti-pattern): Codebase size as progress metric

Sometimes (though fortunately rarely) QA or project management is taking codebase size and LOC as a progress metric to see what the project’s state is. The more lines of code have been written the closer the project is seen to have been completed. This is a definite anti-pattern for following reasons:

• It is extremely difficult to estimate, how much code will be necessary for a certain scope or a set of requirements. This implies that project or product management cannot know, how much code is missing to mark the requirements as done.
• It is more about quality as of quantity of code. Well structured code with avoidance of duplication tends to have less lines of code.
• It is very important and valuable to throw away dead code (code which isn’t used or executed anywhere). Using lines of code as a progress metric would mean this important refactoring will cause a negative project progress.

Good: Codebase size as compexity metric

With a higher LOC metric you are likely to face following problems:

• Increase of feeback time: It takes longer to build deployable artifacts, to startup application and to verify implementation behaviour (this both applies to local development and CI servers).
• Tougher requirements on development tools: Working on large codebases makes the IDE often run less smoothly (e.g. while doing refactorings, using several debugging techniques).
• Code comprehension: More time has to be spent for reverse engineering or reading/understanding documentation. Code comprehension is vital to integrate changes and debugging.
• More complex test-setup: Bigger codebases tend to have more complicated test-setup. This includes setting up external components (like databases, containers, message-queues) and also defining test-data (the domain model is likely to be rich).
• Fixing bugs: First of all exposing a bug is harder (see test-setup). Further more localization of bug is tougher, because more code has to be narrowed down. Potentially more theories exist to have causes the bug.
• Breaking code: New requirements are more difficult to implement and integrate without breaking existing functionality.
• Product knowledge leakage: Bigger codebases tend to cover more functionality. The danger increases, that at some point the organization loses knowledge which functionality the software supports. This blindness has very bad implications on defining further requirements or strategies.
• Compatibility efforts: The larger a codebase the more likely it is that it already has a long lifetime (codebases tend to grow over the years). Along the age of software down-compatibility is a constant requirement, which increases (a lot of) effort.
• Team size + fluctuation: Bigger codebases tend to have been touched by a big size of developers, which can cause knowledge leakage. Due to communication complexity, each developer only knows just a little part of the system and does not distribute it. Even worse due to team-size fluctuation is likely to be higher and knowledge gets completely lost for company.
• etc. …

Quantification of LOC impact is hard

Above statements are more qualitative and are not quantifiyable, because the exact mapping of a certain LOC number to a magic complexity number is unfeasible. For instance there are other criterias which have an impact on the complexity of a software system, which are independent of LOC:

• Choice of programming language/system: Maintaining 1.000 LOC of assembly is a complete different story as doing it with 1.000 of Java code.
• Problem domain: Complex algorithms (e.g. to be found in AI or image processing) tend to have less lines of code but still are complicated.
• Heterogenity of chosen technology in your complete source-code ecosystem: E.g. using 10 different frameworks and/or programming-languages and making them integrate to the overall system harder as concentrating on one framework.
• Quality and existence of documentation: E.g. Api-interfaces aren’t documented or motivations for major design decision are unknown. From developers point of view such a system is effectively more complex because a lot of effort has to be spent in reverse engineering.
• etc. …

Conclusion

The metric LOC representing codebase size has a big impact on your whole software development cycle. Therefore it should be measured, observed and tracked over time (also by subcomponent). Apart from showing you the current state and evolution of your codebase from historical point of view you can also use it proactively for future:

• Estimation/planning: When estimating features take the LOC metric has influence criteria. The higher the LOC the more complicated it will be to integrate feature.
• YAGNI: Take YAGNI (“you ain’t gonna need it”) principle to the extreme. Only implement really necessary features. Do not make your software over-extensible and as simple as possible.
• Refactor out dead code: Being aware of LOC as a complexity metric, you can create a culture of dead-code awareness. Throw away as much unused code away as you can.
• Refactor out dead functionality: Software products often are unneccessarily overcomplex. Also push business towards are more simple product strategy and throw away unused features and achieve a smaller codebase.