How I almost deployed a security disaster and lived to tell the tale

So, I have a confession to make.

Last month, I was feeling pretty good about my microservices project. Six services, JWT authentication, the whole nine yards. Architecting, designing and implemented for almost a couple of years, trying to make it perfect. To think evey apect that might go wrong. And now … I was ready to show it off to the world. Then, on a whim, I decided to run a security scan before going live.

Big mistake. Huge.

The results were… well, let’s just say I spent the next three weeks in a cold sweat, frantically googling “how to fix security vulnerabilities” at 2 AM.

The Moment I Realized I Was an Idiot

You know that feeling when you’re watching a horror movie and the character is about to open the door to the basement? That’s exactly how I felt when I saw the security scan results.

47 vulnerabilities. Forty-seven.

I found hardcoded passwords in my config files (because who has time for environment variables, right?), CSRF protection that I’d disabled “temporarily” six months ago, and JWT tokens that were basically just fancy-looking strings with zero actual security.

It was like finding out your “secure” password was actually “password123” and you’d been using it for everything. Literally everything. Feeling shame right now, by the time I write those lines…

The Great Security Cleanup

So I spent the next week fixing everything. Or let’s admit it, almost eveyrthing, until I find anything else.

Below you will found all my findings and how I fixed them:

The Hardcoded Password Disaster

First up: passwords. Everywhere. In plain text. In my config files. Because apparently, I thought I was the only person who would ever see my code. Thank God, by the time I actually run the security scan, I was as I haven’t pushed it on github.com then.

# This was literally in my application.yml files
spring:
  datasource:
    username: flagship_user
    password: flagship_password

But still, I mean, come on. I might as well have put a sticky note on my monitor that said “password: flagship_password.”

The fix was embarrassingly simple:

# Now using proper secret management
spring:
  datasource:
    username: ${DB_USERNAME}
    password: ${DB_PASSWORD}

Environment variables. Who knew they were a thing? (Everyone. Everyone knew.)

The CSRF Catastrophe

Next up: CSRF protection. Or rather, the complete lack thereof.

I had this line in every single security configuration:

// This was in EVERY security config
.csrf(csrf -> csrf.disable())

I disabled it because… well, I don’t actually remember why. Probably because it was causing some error and I was too lazy to figure out the proper way to handle it. Classic senior developer move.

The fix wasn’t as scary as I thought it would be, as you can see below:

@Bean
public SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
    return http
        .csrf(csrf -> csrf
            .csrfTokenRepository(CookieCsrfTokenRepository.withHttpOnlyFalse())
            .ignoringRequestMatchers("/api/auth/**") // Only for auth endpoints
        )
        .sessionManagement(session -> 
            session.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
        .build();
}

Turns out, you can have CSRF protection AND stateless JWT authentication. Who would have thought?

The Input Validation Fiasco

This one was particularly embarrassing. I had endpoints that would accept literally anything you sent them. Ok, not everything, but they could accept a wide variety. Let;s have a look on the below example:

// This was accepting ANYTHING
@PostMapping("/users")
public User createUser(@RequestBody User user) {
    return userService.save(user);
}

Want to create a user with an email that’s just “not-an-email”? Sure! Want to send a first name that’s 10,000 characters long? Go for it! Want to inject some SQL? Be my guest!

I was basically running a “please hack me” service. (Guys … If someone did it , please share the security results with me too <3 )

The fix involved actually using the validation annotations that Spring Boot provides out of the box:

public class User {
    @Email(message = "Email should be valid")
    @NotBlank(message = "Email is required")
    private String email;
    
    @Size(max = 50, message = "First name must not exceed 50 characters")
    private String firstName;
    
    @Size(max = 500, message = "Address must not exceed 500 characters")
    private String address;
}

@PostMapping("/users")
public ResponseEntity<User> createUser(@Valid @RequestBody User user) {
    // Now properly validated!
    return ResponseEntity.ok(userService.save(user));
}

The @Valid annotation. It’s magical. It actually validates your data before it gets to your service layer. Revolutionary concept, I know.

Tokens Aren’t Just Pretty Strings

Now we get to the really embarrassing part. My JWT implementation. The thing that from the whole application I was trusting most…

I was basically doing this: “Oh, you have a string that looks like a JWT? Sure, come on in! Welcome to the party!”

No issuer validation. No expiration checks. No signature verification. I was basically running a “bring your own token” service. Cute !

The security scanner was not impressed.

Here’s what proper JWT validation actually looks like:

@Bean
public JwtDecoder jwtDecoder() {
    NimbusJwtDecoder jwtDecoder = NimbusJwtDecoder
        .withJwkSetUri(JWT_ISSUER_URI + "/protocol/openid-connect/certs")
        .build();
    
    // Add issuer validation - this was missing!
    OAuth2TokenValidator<Jwt> issuerValidator = 
        JwtValidators.createDefaultWithIssuer(JWT_ISSUER_URI);
    jwtDecoder.setJwtValidator(issuerValidator);
    
    return jwtDecoder;
}

@Bean
public JwtAuthenticationConverter jwtAuthenticationConverter() {
    JwtGrantedAuthoritiesConverter grantedAuthoritiesConverter = 
        new JwtGrantedAuthoritiesConverter();
    grantedAuthoritiesConverter.setAuthorityPrefix("ROLE_");
    grantedAuthoritiesConverter.setAuthoritiesClaimName("realm_access.roles");
    
    JwtAuthenticationConverter jwtAuthenticationConverter = 
        new JwtAuthenticationConverter();
    jwtAuthenticationConverter.setJwtGrantedAuthoritiesConverter(grantedAuthoritiesConverter);
    
    return jwtAuthenticationConverter;
}

Turns out, JWT tokens are supposed to actually be validated.

The Authorization Awakening

And then there was authorization. Or rather, the complete lack of it.

My system worked like this: “Oh, you have a valid JWT? Great! You can do literally anything you want. Delete all users? Sure! Access admin functions? Why not! Modify other people’s data? Go for it!”

I was basically running a “everyone is admin” system and not a less privilege principal.

Making up my mind I ended up with something like this:

@RestController
@RequestMapping("/api/users")
public class UserController {
    
    @PostMapping
    @PreAuthorize("hasRole('ADMIN') or hasRole('USER_MANAGER')")
    public ResponseEntity<User> createUser(@Valid @RequestBody User user) {
        return ResponseEntity.ok(userService.createUser(user));
    }
    
    @GetMapping("/{id}")
    @PreAuthorize("hasRole('ADMIN') or hasRole('USER_MANAGER') or @userService.isCurrentUser(#id, authentication)")
    public ResponseEntity<User> getUserById(@PathVariable Long id) {
        return ResponseEntity.ok(userService.findById(id));
    }
    
    @DeleteMapping("/{id}")
    @PreAuthorize("hasRole('ADMIN')")
    public ResponseEntity<Void> deleteUser(@PathVariable Long id) {
        userService.deleteUser(id);
        return ResponseEntity.noContent().build();
    }
}

The @PreAuthorize annotation. It’s like having a bouncer at the door who actually checks IDs. Revolutionary.

The Network Security Reality Check

And then there was the network security. Or rather, the complete absence of it. And here is the interesting part :

My services were basically shouting at each other across the network in plain text. “Hey, here’s all my data! Hope nobody’s listening!”

No encryption. No network policies. No mutual TLS. I was basically running a “please eavesdrop on my services” system.

# Kubernetes Network Policies
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: microservices-network-policy
spec:
  podSelector:
    matchLabels:
      app: microservice
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-gateway
    ports:
    - protocol: TCP
      port: 8080

Network policies. They’re like having walls between your services instead of just hoping nobody walks through the open doors.

The Numbers Don’t Lie (And They’re Actually Pretty Good Now)

So after a couple of weeks of frantically fixing everything, I ran the security scan again. Here’s what happened:

The OWASP ZAP scan went from 47 vulnerabilities to 0 critical and 2 low-severity issues. SonarQube went from 23 security issues to 0. All vulnerable dependencies were updated.

I went from “please hack me” to “actually secure.”

Lessons Learned the Hard Way

If I could go back in time and give my past self some advice, here’s what I’d say:

Security isn’t something you add later. It’s not a feature you bolt on at the end. It needs to be part of every decision you make, every line of code you write, every configuration you set.

Spring Security’s defaults exist for a reason. When you disable CSRF protection or skip input validation, you’re not being clever - you’re creating vulnerabilities. The framework authors are smarter than you (and me).

Hardcoded secrets are never okay. Not even “temporarily.” Not even “just for development.” Use proper secret management from day one. Environment variables exist for a reason.

Run security scans regularly. Don’t wait for a breach. Don’t wait until you’re about to go live. Scan early, scan often, and fix issues as you find them.

The Tools That Actually Work (And Saved My Sanity)

For those of you who want to know what I used to fix this mess:

• OWASP ZAP - Found all those vulnerabilities I didn’t know existed
• SonarQube - Made me feel bad about my code quality (in a good way)
• Trivy - Scanned my containers and found even more issues
• Kubernetes Secrets - Finally learned how to manage secrets properly
• Spring Security - Actually configured it properly this time
• Keycloak - For identity management that actually works
• Network Policies - Because apparently services need walls between them

Why This Actually Matters

Here’s the thing. Proper security isn’t just about preventing disasters. It’s about building systems that people can actually trust. A way to ensure that the data that your customer trust you, are being processed in a way far more safe, than he expected.

When your secrets are properly managed, your developers don’t accidentally commit passwords to Git. When your input validation is solid, your support team doesn’t get flooded with “invalid data” errors. When your authorization is granular, your business users can work efficiently without security getting in their way.

Plus, auditors love clean code. Insurance companies love good security practices. And customers love knowing their data is actually protected.

It’s not just about avoiding the bad stuff. It’s about building something that’s actually professional.

Why I’m Actually Glad This Happened

This security nightmare was actually the best thing that could have happened to me, at the right time.

It forced me to learn about security properly. Not just the theoretical stuff, but the real, practical implementation details that actually matter. It made me understand why Spring Security has all those “annoying” defaults. It taught me that security isn’t just about preventing attacks, but it’s about building systems that are actually trustworthy.

Now I can sleep at night knowing my services aren’t broadcasting passwords to the world. Now I can confidently show my code to other developers without cringing. Now I actually understand what all those security annotations do.

What’s Next?

This whole experience taught me that security is never really “done.” It’s an ongoing process. Every new feature is a chance to implement security properly from the start. Every security review is a chance to learn something new.

And if you want to see how I actually implemented all this security stuff properly, I’ve open-sourced the entire project on GitHub. It’s a good example of how to build secure microservices without making all the mistakes I made.

Thanks for reading, and remember… The best security vulnerability is the one you find and fix before anyone else does.

Check out the complete secure microservices project: github.com/mikopos/springboot-flagship

#Cybersecurity #SpringBoot #Microservices #Security #Java #SoftwareEngineering #DevSecOps #CloudSecurity #CareerGrowth #TechLeadership