From 9fd334c47fff05e4aeb6ed738b152b260c3f2670 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Mon, 9 Dec 2024 22:53:32 +0530 Subject: [PATCH 01/12] Create chapter08.adoc Chapter 8: MicroProfile Fault Tolerance --- chatper08/chapter08.adoc | 1 + 1 file changed, 1 insertion(+) create mode 100644 chatper08/chapter08.adoc diff --git a/chatper08/chapter08.adoc b/chatper08/chapter08.adoc new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/chatper08/chapter08.adoc @@ -0,0 +1 @@ + From 6a3d751ab1a442458f189aae373d8e4ace61d8cd Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Mon, 9 Dec 2024 23:01:06 +0530 Subject: [PATCH 02/12] Update chapter08.adoc Added chapter Introduction --- chatper08/chapter08.adoc | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/chatper08/chapter08.adoc b/chatper08/chapter08.adoc index 8b137891..540d6183 100644 --- a/chatper08/chapter08.adoc +++ b/chatper08/chapter08.adoc @@ -1 +1,19 @@ += Chapter 8: MicroProfile Fault Tolerance +Microservices architecture involves breaking down applications into smaller, autonomous services, improving development flexibility, agility, and scalability. However, this design approach does bring in additional complexity, particularly regarding fault tolerance. *MicroProfile Fault Tolerance* tackles this issue by providing a mechanism to ensure resilience and maintain reliability despite failures. + +This chapter explains how to enhance the resilience and reliability of your microservices using *MicroProfile Fault Tolerance* and demonstrates how to implement key strategies such as retries, circuit breakers, timeouts, and fallbacks. By the end of the chapter, you will have a thorough understanding of how to ensure the resilience and stability of your microservices architecture. + +== Topics to be Covered +- Introduction to Fault Tolerance +- Strategies (Retry, Circuit Breaker, Timeout, Fallback) +- Implementation Retry Policies and Configuration +- Avoiding Cascading Failures +- Configuring Circuit Breaker +- Setting Timeouts +- Implementing Fallback Logic +- Isolating Resources for Fault Tolerance + +== Introduction to Fault Tolerance + +Fault tolerance is the ability of a system to continue operating correctly even in the event of unexpected failures or errors. A fault-tolerant system is one that can detect, isolate, and recover from errors without human intervention. This resilience is critical for maintaining reliability and consistency in distributed or complex environments, especially in modern microservices architectures where individual component failures are inevitable. From 8c35bb7ffa51d946fb131956766837681ab9cfd9 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sat, 11 Jan 2025 22:29:49 +0530 Subject: [PATCH 03/12] Update chapter08.adoc --- chatper08/chapter08.adoc | 243 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 235 insertions(+), 8 deletions(-) diff --git a/chatper08/chapter08.adoc b/chatper08/chapter08.adoc index 540d6183..dd1c4f9d 100644 --- a/chatper08/chapter08.adoc +++ b/chatper08/chapter08.adoc @@ -1,19 +1,246 @@ = Chapter 8: MicroProfile Fault Tolerance -Microservices architecture involves breaking down applications into smaller, autonomous services, improving development flexibility, agility, and scalability. However, this design approach does bring in additional complexity, particularly regarding fault tolerance. *MicroProfile Fault Tolerance* tackles this issue by providing a mechanism to ensure resilience and maintain reliability despite failures. +In Microservices architecture, an application consists of multiple smaller, autonomous services. It enhances development flexibility, agility, and scalability but introduces new challenges, particularly in ensuring the application's reliability and managing failures. Unlike monolithic applications, where defects are localized, a single failure in one microservice can propagate across the entire application, potentially causing widespread outages. Therefore, fault tolerance is critical for microservices architecture to ensure that failures are seamlessly isolated, managed, and recovered. -This chapter explains how to enhance the resilience and reliability of your microservices using *MicroProfile Fault Tolerance* and demonstrates how to implement key strategies such as retries, circuit breakers, timeouts, and fallbacks. By the end of the chapter, you will have a thorough understanding of how to ensure the resilience and stability of your microservices architecture. +*MicroProfile Fault Tolerance* offers strategies for building resilient and reliable microservices, ensuring service continuity and stability even during unexpected failures. + +This chapter explains how to enhance your microservices' resilience and reliability using MicroProfile Fault Tolerance capabilities, detailing its annotations. We will also demonstrate how to implement key strategies such as timeouts, retries, fallbacks, circuit breakers, and bulkheads to handle faults. By the end of the chapter, you will understand how to use these strategies to enhance the resilience of your microservices. == Topics to be Covered -- Introduction to Fault Tolerance -- Strategies (Retry, Circuit Breaker, Timeout, Fallback) -- Implementation Retry Policies and Configuration -- Avoiding Cascading Failures +- What is Fault Tolerance? +- Key Strategies for Enhancing Fault Tolerance +- Implementing Retry Policies and Configuration +- Avoiding and Managing Cascading Failures - Configuring Circuit Breaker - Setting Timeouts - Implementing Fallback Logic - Isolating Resources for Fault Tolerance -== Introduction to Fault Tolerance +== What is Fault Tolerance? + +Fault tolerance is a system's ability to continue working correctly even in case of unexpected failures. A fault-tolerant system should be able to detect, isolate, and recover from errors without human intervention. It is critical in applications based on modern microservices architectures where individual component failures are inevitable due to network issues, resource limitations, or transient errors. + +== Key Strategies for Enhancing Fault Tolerance + +Some of the key strategies for enhancing the fault tolerance of a microservices-based application include: + +=== Timeout + +A timeout sets a time limit for operations, preventing indefinite waits and freeing up system resources for other tasks. For instance, a timeout in payment service ensures that the application can recover gracefully if the payment processing is taking too long to respond. + +=== Retry + +A retry allows the system to automatically retry failed operations, particularly useful for handling transient errors like temporary network glitches. You can customize the retry policy with parameters such as the delay between retries and maximum retries. Adding jitter prevents synchronized retries across services. + +For example, a payment service can retry a failed payment authorization request with an external payment gateway to ensure successful transaction processing. + +=== Bulkhead + +A bulkhead isolates failures in one part of a system from other parts by segregating resources, such as thread pools, connection pools, or memory, among different microservices interactions. + +For example, in an e-commerce application, the catalog service can implement bulkheads using separate thread pools or connection pools for different upstream dependencies, such as the product database and the pricing service. If the pricing service becomes slow or unresponsive, its dedicated resources (thread pool or connections) prevent it from consuming all the resources of the catalog service, ensuring that requests to fetch product details from the database continue to work unaffected. + +=== Fallback + +A fallback provides a default response if an operation fails. It ensures the system continues providing a meaningful response instead of completely failing. For example, if the database fails or becomes slow in the product catalog service, the system can fetch cached product data to continue serving user requests for product listings. + +=== Circuit Breaker + +A circuit breaker stops an application from making too many unsuccessful requests to another system. If the number of failures exceeds a threshold, the circuit breaker will "open," causing all subsequent requests to fail immediately. After a configured delay, the circuit breaker will "half-open" and allow limited requests. If those requests succeed, the circuit breaker will "close" and let all requests go through. + +==== Example of Circuit Breaker States + +For example, a circuit breaker can be applied to calls to an external inventory service in the Product Catalog Microservice. If the inventory service starts failing or becomes unresponsive, the circuit breaker will "open," preventing repeated requests and reducing load. After a configured delay, the circuit breaker will "half-open" to test the availability of the inventory service with a few requests. If those succeed, the circuit breaker will "close," resuming normal operations. + +=== Asynchronous Execution + +Asynchronous execution allows operations to run in a separate thread. It means the caller does not have to wait for the operation to finish, making the application more responsive. For example, when a user searches for products in the product catalog service, the service can asynchronously fetch product recommendations from an external API while immediately returning the main search results to the user, ensuring a fast and responsive experience. + +When applied individually or in combination, these strategies form the foundation of a fault-tolerant microservices architecture. The following sections delve deeper into their implementation and best practices. + +== Fault Tolerance API + +The Fault Tolerance API equips developers with annotations to enhance the resilience of microservices against failures. It integrates seamlessly with the MicroProfile Config API, enabling the dynamic configuration of fault tolerance behaviors without modifying the application code. This section will explore using the Fault Tolerance API to build a robust, fault-tolerant microservice. + +=== Adding Dependency for Fault Tolerance API + +To use the Fault Tolerance API in your project, include the following dependency in your `pom.xml` file. Ensure you specify the version (e.g., 4.1.1) compatible with your MicroProfile runtime. + +[source,xml] +---- + + org.eclipse.microprofile.fault-tolerance + microprofile-fault-tolerance-api + 4.1.1 + +---- + +The Fault Tolerance API defines a contract for fault tolerance implementations. + +== MicroProfile Fault Tolerance Annotations + +The MicroProfile Fault Tolerance Annotations provide a declarative way to implement fault-tolerant behavior in Java methods, allowing developers to handle failures gracefully with minimal code changes. + +=== List of Annotations + +|=== +| Annotation | Description +| `@Retry` | Specifies that the annotated method should automatically retry on failure. Parameters such as `maxRetries`, `delay`, `maxDuration`, and `jitter` control retry behavior. Configurations can be externalized using MicroProfile Config. +| `@Timeout` | Specifies the maximum duration (in milliseconds) the method can execute before being aborted. If the timeout is exceeded, a `FaultToleranceException` is thrown. +| `@CircuitBreaker` | Defines a circuit breaker mechanism to prevent repeated calls to a failing method. Includes parameters like `failureRatio`, `delay`, and `requestVolumeThreshold`. +| `@Fallback` | Specifies alternative logic to execute when the primary method fails. This ensures meaningful responses and graceful degradation. +| `@Bulkhead` | Limits the number of concurrent method executions to isolate system resources and prevent cascading failures. +|=== + +=== Implementing Retry Policies and Configuration + +Retries are a fundamental fault tolerance strategy for managing transient failures such as temporary network outages or intermittent service unavailability. The `@Retry` annotation in the MicroProfile Fault Tolerance API provides a simple and effective way to implement retry policies. By customizing parameters such as the number of retries, delay between attempts, and conditions for retries, you can ensure your application responds to failures gracefully and minimizes downtime. + +==== Example: Retry Implementation in PaymentService + +Below is an example of applying the `@Retry` annotation in a `processPayment` method within a `PaymentService` class of the MicroProfile e-commerce project: + +[source,java] +---- +package io.microprofile.tutorial.store.payment.service; + +import org.eclipse.microprofile.faulttolerance.Retry; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.MediaType; + +public class PaymentService { + + @Retry( + maxRetries = 3, + delay = 2000, + jitter = 500, + retryOn = PaymentProcessingException.class, + abortOn = CriticalPaymentException.class + ) + public Response processPayment(PaymentDetails paymentDetails) throws PaymentProcessingException { + System.out.println("Processing payment for amount: " + paymentDetails.getAmount()); + + // Simulating a transient failure + if (Math.random() > 0.7) { + throw new PaymentProcessingException("Temporary payment processing failure"); + } + + return Response.ok("{\"status\":\"success\"}", MediaType.APPLICATION_JSON).build(); + } +} + +class PaymentDetails { + private double amount; + + public double getAmount() { + return amount; + } + + public void setAmount(double amount) { + this.amount = amount; + } +} +---- + +==== Understanding the `@Retry` Parameters + +|=== +| Parameter | Description +| `maxRetries` | Specifies the maximum number of retries. +| `delay` | Sets the time (in milliseconds) to wait between retry attempts. +| `jitter` | Adds a random variation (in milliseconds) to the delay to avoid synchronized retries. +| `retryOn` | Defines the exception(s) that should trigger a retry. Defaults to all exceptions if not specified. +| `abortOn` | Specifies the exception(s) that should not trigger a retry, overriding the default retry behavior. +| `maxDuration` | Limits the total time (in milliseconds) that retries can be attempted. +|=== + +==== Best Practices for Retry Policies + +- **Limit Retries:** Avoid setting `maxRetries` too high, as excessive retries can overwhelm the system or cause cascading failures. +- **Use Jitter:** Always configure jitter to reduce the risk of synchronized retry attempts by multiple services. +- **Abort Non-Recoverable Errors:** Use the `abortOn` parameter to exclude critical exceptions that retries cannot resolve. +- **Monitor Metrics:** Integrate with MicroProfile Metrics to track retry patterns and adjust configurations dynamically based on real-world performance. +- **Combine Strategies:** For robust error handling, use retries alongside other fault tolerance mechanisms, such as timeouts and circuit breakers. + +=== Avoiding and Managing Cascading Failures + +In a distributed microservices architecture, cascading failures occur when the failure of one service propagates to others, potentially causing widespread system outages. Such failures often result from tightly coupled services, unbounded retries, or resource exhaustion. + +==== Causes of Cascading Failures + +- **Tight Coupling:** Dependencies between services without sufficient isolation mechanisms. +- **Unbounded Retries:** Excessive retries on failing services, overwhelming resources. +- **Resource Contention:** Exhaustion of critical resources such as thread pools, memory, or database connections. +- **Lack of Fail-Safe Mechanisms:** Missing circuit breakers, bulkheads, or fallback logic. + +==== Strategies to Prevent Cascading Failures + +- Use **circuit breakers** to isolate failing services. +- Apply **bulkheads** to limit the scope of failures and resource usage. +- Set **timeouts** to prevent long-running operations from blocking resources. +- Design retries with care to avoid overwhelming the system. + +=== Configuring Circuit Breaker + +A circuit breaker is a critical fault tolerance mechanism that protects a system from repeated failures of a dependent service. It stops repeated calls to a failing service, allowing it to recover. + +==== Circuit Breaker Parameters + +|=== +| Parameter | Description +| `failureRatio` | Specifies the proportion of failed requests required to open the circuit breaker. +| `requestVolumeThreshold` | The minimum number of requests made in a rolling time window before the failure ratio is evaluated. +| `delay` | The time (in milliseconds) the circuit breaker remains open before transitioning to the "half-open" state. +| `successThreshold` | The number of consecutive successful test requests required in the "half-open" state to close the circuit breaker. +| `failOn` | Specifies the exception(s) considered failures contributing to the failure ratio. +|=== + +==== Example: Circuit Breaker Implementation + +[source,java] +---- +@CircuitBreaker( + requestVolumeThreshold = 10, + failureRatio = 0.5, + delay = 5000, + successThreshold = 2, + failOn = RuntimeException.class +) +public String getProduct(Long id) { + // Logic to call the product details service + if (Math.random() > 0.7) { + throw new RuntimeException("Simulated service failure"); + } + return productRepository.findProductById(id); +} +---- + +In this example: +- The circuit breaker opens if 50% of requests fail (`failureRatio = 0.5`) after at least 10 requests (`requestVolumeThreshold = 10`). +- It remains open for 5 seconds (`delay = 5000`) and transitions to the "half-open" state to test recovery. +- Two consecutive successful requests (`successThreshold = 2`) in the "half-open" state close the circuit breaker. + +==== Best Practices for Circuit Breaker + +- **Set Realistic Failure Ratios and Thresholds:** Tailor parameters to your services' expected load and failure behavior. +- **Monitor Metrics:** Use MicroProfile Metrics to monitor circuit breaker state transitions. +- **Combine with Other Strategies:** Use circuit breakers alongside retries and timeouts for a robust fault tolerance setup. + +=== Combining Fault Tolerance Strategies + +Combining fault tolerance strategies, such as `@Timeout`, `@Fallback`, `@CircuitBreaker`, and `@Retry`, ensures resilience and efficient resource usage. Externalize configurations with MicroProfile Config for flexibility across environments. + +== Summary + +This chapter explored the MicroProfile Fault Tolerance API and essential fault tolerance strategies: +- **Retries:** Automatically reattempt failed operations for transient errors. +- **Timeouts:** Define maximum execution times for operations to avoid resource blocking. +- **Circuit Breakers:** Prevent repeated calls to failing services and allow graceful recovery. +- **Bulkheads:** Limit concurrent operations and isolate resource usage. +- **Fallbacks:** Provide meaningful responses during failures. + +By leveraging these strategies and combining them effectively, you can design resilient microservices that gracefully handle failures, minimize disruptions, and ensure a seamless user experience. + + + -Fault tolerance is the ability of a system to continue operating correctly even in the event of unexpected failures or errors. A fault-tolerant system is one that can detect, isolate, and recover from errors without human intervention. This resilience is critical for maintaining reliability and consistency in distributed or complex environments, especially in modern microservices architectures where individual component failures are inevitable. From 91408407c1292bfda036d4e864ac252990fe54f7 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sat, 11 Jan 2025 22:38:38 +0530 Subject: [PATCH 04/12] Add files via upload Figure showing flowchart for Circuit breaker states --- images/figure8-1.png | Bin 0 -> 38427 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 images/figure8-1.png diff --git a/images/figure8-1.png b/images/figure8-1.png new file mode 100644 index 0000000000000000000000000000000000000000..122a31d29f0c7dd344de3c25349f43adf61de7cc GIT binary patch literal 38427 zcmcG$1yoi0*EYHV0cn-q2v{hcTTqY^=@w}uC8b*$6i`Z0LFq1OBn(nY3E70wpdck3 zQX<^Bz;oXJ`+ncJcZ~1ed&W2e+^oIUFJ?URna^)MQd5y7BSsUWP$)8axf?f8DBKnJ zzu`D8d{R1r%K|^xU8Qwg%^V!>*_vCpqGZkO&7DnL&F`Esy?4gS)z#5On1{#F*2Lb` z&CZtF%)!q6a~m@Xg_CT3OULz}&rvvVAJ0U$XjO+Hi4)DszuBg($Zwo=&iI&XgD%lm zr&LpN9`_>DaT85gVm?*tYkBq}t2ztwSBJ5^v5uueik{#BdZ#N7CvXmAHX5YN7f43* z`{m6x&D1fH1P_XN^JtZ?GLqDU6g(Ze5z5roPov&o(=M)mx33E40oSQJ-Wzy0jz0}K z+kZQqQn~8UQt7Y8!guyk?C>HOc7VuhRg=NBWk*1d0&d9%)QSX?g@Vf0KCo77R< zklLXdZK{-B?(g1 zze&0Bg4IzXJ0#=Jh=CF(zVgXuJ6GHt#N&NSx+Y?~=9-&(MTI4ygD6x0O8$oAElS?#AlfD^96X$a+?Lt2%SwB>rm8oSZ2RE_*gEdv0DUr!Na`*o!6-cGGW> zmmip8M6|N&mK~VsLoF>{=o-*iY_0F#?_Ut;`utf;pljX7-NQZ41&jnGG5K=^AAX|r z?n~gokND94lOL%#b$e@y&dzJ0%zgv;+C0q6DFFci_4V~vU8eU|8Yz4_w@z~@y{rKdp>|>^>WddW%6ILjcO|g;xpUbSQ4>h*%;dC4hp5go5VU&d=^7Y7-Mz6oWW~~%cp!%ecPqS9K*l>=1 zt?`39r>33}d;9U)dWXQHN00WBa2ALT$K<{|R^vAR)op1cPL*#egp}TSV^)3N9_3B| zQDI2c%XiDk*h*{CL7a0go(ZvhE7bh%7S6 zKO0^(#l_2QF&u-+eAvvK9B*&$-l0%UoCRu_T>!x`eEhd>-*OpNZ~NU#O~dq-nYUMs z`y~_;8^1ty33G;nqy5X5m~>Ke^sb_H%joFn;9y+a&Og&)@cQ-hOP1mb8iMB*vksx; zjN1yRO=4ui4CxsegcM92xMWUFPCI++T|58&Wy8yNxYEwo$44jC$;G8{svdO?S%S{v z#KdS}S5^|to6~d>hc^)@D=T~Fv8-a9t(mX&@285$Lvdf=;e9-VMWL+ZU0pHwXajQ6 zdStU1HfB=%_cwY~;jtzrCgHYa2i9q&jI65v4JglRj zfYUGeSYg`zD|f#ZG=@^Fx6;)ghk2BoH1hpLUGrwm6?@gH&_-d{u4!g7K;j~jERlqHhA+S z1+SsKp`;XFpQ9jXDw`m-l^bju!yZ=z0tH>8?)ER7IRP|R40 zmq=4t%{s1Iq9hO#!z)*d}djT{-gl5#5jWt&?R_s>d&_-qbZ z!d!eJf6`J@$E2imy+EH`;&u7)@g%#ei_37GzqrvCpYqjyLvQ5{%Hb*pW^q5CX7(@F z-yLZBaUaJR<03N~fSK*ctG4v_du%QYdT%W$%gH%9ImHQBk{f00c2B-a`CuKIqnxm* z{5D^^nE3Gt+$-g)@7(6YBzd{Hi_BW1sPk`;97E^YA1($vo-&w+J~PLrF9!>z<_!nYkAS5hVKe`su^q@)zG8)|C19~7iha^_ZYVxp}rNznHc z+1mBbB6tL(+u!a}I6FJ1r>CO@ZLio4lsuxmv@}+uZzx%P{`{y(G#zeOc(|056hu_F zp`lODF6G_nFS?DGL%vMDR9*6n1og* z?Hcw13rly#>r-X-)~8>-q+87fcj83?GNxlhLcpdkcmL1t*X9amAuNsitSkHY_=MAN zStVzG&^EjmY_S1m@%bg+WL{2kI$NBk-$H5Y=M?d-SnJvpG2eHkTxwLFyK_1Hh5A(< z%VRMd+Sv>kyCK(4PgyXQaDkp-pq<+_!&q_|`+10aK;v-z>EKl`j#zBO$0x3+Uq zGdMU%b-s!DRA$9>hiB!RC;|4jS@C8}gu_3r;f$rz2!|P(R8B*ccJn`f zqUq?~m^6lfaff>(@|pxalE`?2ub1CH4_RNiVHPra&ZyaoaRt0cQ+zf`=BwSWQ!7lr zUrO=$AV&1Fzu4%`oePVLmMOl1etX+r^0hVkj+2x7c^)`(VzD0*9BQxLTf4)Dr5omF zp_}uTa}_v${ytaTY74_e|MbDHVt+Dv{s&xLFZu2#J9H(Bg6V52=z*uxMjVR=+o|2? zQ7q?m{vr2{R{t(|`@fs?aOd0PWAUZoDjqHf!J!vNiGLk4nfaWA=?@bMrC{bJKob&% z3Rr$Fnc_2RQA<@H37Q4N`Sj@%)#HKRr`N)R-V)!uh>JK=Vp3A`2l;0gYYz5*m!Du_ zQo+g6$WcM&E+fjii>Bk0Yry^W||;AC@APs zvS(0G`>*+aNR-+IIt^`e7GRg^>fODn;x*>&H0){HJ-O=V`LNqNJEf|IpOZvnDFyMt z=t_tu2!#%|TgAtGw{GjDek=3)5IxcckW#K)u+K3 z36Vs#JoI_}kL5qdpGrv0rp@h?NtoHT!$5q0Z>lrekm(t|wopzIc2hcrZg`TYS7FmU zHu8$mcgWo3goJ$yum3poEW@2K(;P!1cmI8yfH%o(_Kax04=XbTG zxjA^Vsk=LRF~oSgmb&?&$Jo8l(T%Mw|J7#N=HCkPcx2KXZK4y>4Pp9%LRf4vDH`*z zQYs+}pP8AtH*k8kJA=wA_uim#@4p^!S-ikf&(CikqN~Hmo3pXlwa4Q7IyY~wb;~k8 z3Y6jX5xYL+Rb$9$@xXJvx)n9hI*sAPD+r+p7f2tW6 zT=U*s07uF9NgzH*Jf(zS1hV0`E;E66gxh%(w(2r6Ent`10;tr}hxrZSZJz-xsh}~$ zP}es*|9q~0AiFOe%%lE*zhSD8X3a)f0Fz?Ha0d0 zIeVv%SDFHr(yz4r{#aN-8Z6P61k%&y(rA0vgH+#f->uOXMt+}|9Z>ZDKod<{TTXH2 zpx9>giY!_J!MFoa1qB6=Uwe%FO#gR)hOG{KDTIt+x-&txctLZ2SFDf8_2X=P_xIs) z>s}}>b}~qek+}_|Lb|4=qH2O|4-V!kyUL|k{w=(mr{j#6STgx(|0_7R5IB#eBgXig z3>f|Y69M zDHa;@y#L>jFMyum;RI(=z>o7YGcyYdiBF!?YvY$FB3~cE)_B8rS+V5~9F5jgQ^TwS=Y-QVD@ddZ2 z|Nb@cON|+C9lw7-e6f-smjJovmi5@5pZUexgLIr{FJ26Zf8pSSQberkO&xfRPflL^ z^XJb2H?MM$8(YH{jF(}BYFy`T`Y{o->(=^Q^>d+k#ef*@d%NLE*sq5v4;q}Km_!vK zJE}wiq!DOb(>N^fW$|mFetRs}I!v~Hj0*sFN?=1etlA?9yf0+184I^>+&CtV z*ADA?2euqGcx-H}xMp)`tj0T!mz|wmTs);%FFIaWvl2EDg2TJ_@2l?i+xO+F)6>)6 zk|P0s_R*9asI(ha{BLWv`U!&%V3@evakA*O$KRZBIVl zS{~=IZv|j*h;@yZ*~u_M_9IqGV) zAuclX8T`GLt+nSAM~bMoU~n=dI@LJ-=7xr|7cSh`)d`E`zTNik!sTPIcPuUY^E5(# zpZpGlLD+FXS_xO2CYDCu=iPzd0jo1!aXh2E4yZ}3KtoH5hMxYa&-RMbWXs}kmFhdm z*J{Mp-RaUgIxovhGBa7Etram%5f@) z)A9IJ3<(KIR&a*w<7@*Kx_jzp)WP5CN8H6teZ7<_SY^@M4UP$5;x~@RtG;-H9AfK zBnK3*4g1HDmyQY~`0LaPc%6`kzZM&*o|WJ=@D#D=c*!K{g?0YDxtN#uWe^I`DbyzS z$N0;4!Lu51Bg@GP-8?*uN=<`31uk6*<uO`(ty_E56MIs;cu`If>Tc+yW)#db0Q;FkLioe|x5rI~+B>Rax4J^jX?N%ow01oaEJ-|Iy> zZ}$;**0|E$Tl-aYe+vEn?3O2~Bnd-;E_Kk31DNrzr-Zx(Vx`oqm8wqXLb{{l=7MhR zt&8H~2R}YNquDtaqq?Y)c*SXHorQwZa|8XqCnYv+2qKgJa=AUwJYYIcXY?~3az7ToJ~S=lxlx-Ts|;qqNAcXz?@GSnA@bqm;PW%joP zR19TnX=$m=ehkhD8&(@`n{udM-+L^t*%7OCA~$*tD5)4B7{$a2Vj-sC8{O&!!XYmW zfdDecN@x&V5Sb>?`2Ut73>Z?$>6LeI4h+#JF-}1qZS%gVsVO3YbTu1;Rl8b9?SMPL z1y6$ui~E1??uNz5kYa9KpDN!p(fkWuaKNBp$ixFm&cMJx zugYHeY6vdbpI__iF7EDz@7=kKsOz~Q%%H3qUAjbN<5~bY;VOfxLDrvJTKcSwdxW}% zPB1e50RSQ_BBGH>f6bHkPZF;Qt`Ska;o+C>8y0V^t=XJ)n(JYyJrz)cKsam!*n5Tj zXna;2PF#EF;heucXpOQ<|#0jCY>{F2DC0+jGxAu%-a z#KQT`0g5|lUJ|gWz6kMyy>&?N6P>WzVN)-7jP{|9ZPI*T+6JtsRZO{>f<%0&bc_Ys zJ$cskU0E5w^$rl~ep7AKb3_D!6fi0qnHx8nz{LjnJM}x`FGITC1hkQIZ#oWKwZgjB z05nS7RhmA&uo1`#PK!chh*Kc{?RVf$iE2}-ymSRJ4M+6M>kO|``o@CpikZETzb zhB4cPO+@7VglN453|*oV>LGVvuTDs{fB*hnQc@D*R$=oc%+O9l-ozv=A%UT0UqxA& zQP7s|efTza)bjw0*}T#nG%*IMi?raqJs;sDtM8C~Aa`=Hv!_oze@^A^@4v)toDXJ} z+7iJeDl8_p4~qfN=jzp~&u@>6j^@O8|IEt)K1uZx)Pm-Kb>_bozQ4DAu?io@x#OXOjr)t)8$ZVpB5rR}(_7Eg zbNU>&Vl5OD6&2gMWwyUeAvE6CFcO+eqm&c)o8T-FPqxyK6i zh1dtntud37zTgCrYO}v?KZc6}7#S^f^pmdz{4vAowkDNTU@>dIek}qY z$}%+9o8#h^`9|qH1!jwk7Qf2o%PXj6?1su4q#HJo$Y^uZ%4!y(94FUXk7cHskKEkc zG>TTS?>tx2w`&X}rQ=VM@|_b9c*c>rm?+|*qW57e|HOsfLY;HBx%sMJKYj5+TSrHZ z#H&V3NazK1m{q&w=fvnbqO;?aUfex4J&S5r?S25@=sC4x_K5^&I4~8SkGPSnqM}L8 zG35{NPkP<#`&RJ4J3gofkEqO&eVqKQ(xlOGyXqxj#>Vm(eO747VPh42G9eJ-aEYJ)X|f#9tFudB2$o-(vIR@4prKJ) zTibI=+J>KOfC2nMANwJ>6j=fDS>6&ym=E)MB~`w zs~$@Rz1hmm6q!J3Q!^_XdanKW1d#1^wWm7o1)|ru%Frt00z?p0cq%2^__M%N<+FqY zMb)?(oY$~kVUG*YcTM!}+B#I(4Nrgn?zFeyuIpnfF9TzkO@CZ@Tj_`aGZ%)!>DTi^+5zbj__#X#o*SGtTvucRCGd<%7Hq0D;;i3Lc!9rP6BUJJmuv?{7WUX;${;mO zKHkE<@>OoG9NKCCx0yWN!$VQ+GJB2)5JIv1VWFX+FNx@OWL1K`0~H#85_Vk3V3B0Q zvWmr4BT-!E#*>?*m(+Qxyrk7lHR;ajk?4iS?<07iSRU%wu74GZoXWg56$Dkmw4 z^9QQpQ}1_U_8}{~ye>0M-GK%&ut&GUbiv^lf|A!~x->eWU@Ikl?nwp4Am@>DYBP8S zI9)w<>_@1L)|B2ZVs9(?{J3N4kXPo(SH|c55{6_%Sd2a#Qt0IEthkWylfp?wkg}i~ zJwH@2ukTP}eh!Otwr(lU&G;W|`A-3bSMBzFX-P$Y7Dxx6fbpd^AViXqk^*hWEM%Xk z^UMijWoQrVGHh>-b+V`dP@=%y5B5L&TA(}m^XE{GD)Z21ATN()wcg}3sP=RMLG?t=~aP({?!49s|H zTW5bSNwI$qQK#9$d$H%qfs8y7HBN1w~D~l;+y*k_=gW$_Y74s zTVJ@=Ia0MuUc_|hAnU8-aV;Td;iKQ=MW>LD>XECSJ+Yv3b-WBW!^j`vy-CKZuPSGH z2k79Iv)?qK08W*MHa0K|6UuX*vQy<+)z;citZhR?bK-u(1<)c-mG`O{>+jb-z1FlE zrI4HWRjk0qlC>9L5f3TGet&PrJ;7orjrYm!*;Z$mtEzTEwYaK@ zBXKu`obl(>RI{6-hsPFBvg?`+uy7Lj;0-{;12d`EcskN!A&VgfDCTwK3&0cSiu~T$ zdBj)^J_g3sx4QGn7#Xk0{{DV)axy?<%$Rz7nqzO#ZIRkjC@T%X?MTPB*4O*zH_py$ zTO%$$EUX}zNHqcW!qLsmt~Z-3(gUH}>Q0_K2^<_Y+hg4Kh43%H11kPWNl8$$(lIi2 zY6m$(>PKhy&yS9b=smwXSjJLBbF73OI>M&Q7Xh`*egi%qdaEMK)#sx;q(aH3wDAIB z-4=&pel0=G2}B$4(5P}Q=!uVx)&zfFM1ZET8B9qhb#;9mI@#HfcdU}@>ck^G0Cm=1 zL;#lX6>h>Jpxpze3w27<@wo{JO4Jv6j19;Q*uXPLwim0$frG9n7`Zu5Oh%Tq8nwU1 z@rv2Ic4H(De_IA8SFp-qTt!haKYGc+7pSskIA(#YUYI0JN@(h!|%usIp z&u>~SEtF`*e8FUx{jJ+JHa4D~+mOPx{j6UU8P=eIdA9o?L>a&`^42BPuoZ9)q_q3u zJN464)YRRxHeWr#n$)B*CHBLWk;}pEP@9o4Z{Ba7bA1O1vAX)&{sX*(O2D+h1wldhtu~D!Ns+^Vd7Qw z`&dsPLq4Q1Bc1q{mf#OQ+IZ(7bZ$@M@wUP$t|nbH7W=cIvA>sl9BxMns~QyK zPSQ?ON9Uu^xd36ipjE#X-EL1|K^poNI@O+hwIQdds@dDUm zheWTycw>ELNeM6V_q;p_$iw;u1|2WN%4~)TZiD)ugf2dLhe@j&mg&f0rvTUt-7Q`m zeg7gxR$6*$W=6Z9T|*r+}w>%q*VF#Q;JkO`4M@8&XsiKrkcf_)P)=0+@9-TYPkMLw_!5|F7JH zh4DJ0vN8US(Sp($>2~0usOQm(--4ngKiy!?Nf7nk%#3ID-zzO;)Edb?b5~y4<|m!m zE~626iNH(p|KK*~vIg=ra=?VF+br{;7>yt!J9pXb6@Zh=Ko;$R*u=W^FL&7;psA;) zkniK`3*sZ?Np%1@&~%3)u%7r(LvYpULe?6h9_Uqq8$qen&U|AqT{?VQ=9rqSa;m(N zk}V(xD=SvzQb4Y{rKZip4t1T{uaNnO7}R)aBJ&w4cr+tD$_p4oZ}0}8kRWT^?4+%P z4sch`*)&d{J`F~b`Iv=n{GUm|Vj1&g<0-qlyF)|jhq$#uv7FT#dO_QPrbm=>nzS*0 zyZjq#;UN(K-&nu0{ZHM3orWmd7aG8sE{%|blYQ3Y6?m^VH>^HW6}tzO(wpmS><6G$ zVBIP?Xhj(6S3MF^Gwy&i-x%uNym{R}JUpC@ta^WUzAsHB`PwHf*lOl$o{;p#PxIe- z=o!-fek}2-3#$}ezACgeo2;&XI)C-5u3EQAb0=|2RmV$D!=M81_@!?x1T2)k26`&0 zi^cwb=DRawSu`W;pWT(;VZBtWVV&Uw_Q~1enwrK}1Q~<{KqkB`-=y-f!AT44D z(C+rMR8%V*$I<{-A_|2i0snm;2xgl`Tu^v2oj4)E7^Y&ze&@hIYwTB3R|wVovsNZ= zrM*?t_?J1VDcrfUB|VQ5Bkbb%R;!QcPVkLKdcn`3eyWBdedFhi(@%dam-WoEd%blBK&;nXo4{W9>`0NYdfyR^ z&YcP0GM|m7trf8lZn)rY9GueeMr4XHNp`gPxJf(j&xUZ4dO5d>$nV!1RRtsj^?4r+G9V~Ha9qu#O38ZA%aTPvOz+DATf*yK)Ki)5z6)00$ zdkvdAS7HDSb$+vjdwd4dY6YC`jtw7E@1#F=37ih5HH7YeZ0vEFO16)BH^tiI=X0$w z(x3BuJ3s-*R;kWQjXm< ziHz(=qvDZL!jtNc_G(D;juZTXT$E8+T#i$TD+KA?pk$e8c1ZohTT0-6TX6wcyRQwz+< z#IL$FC*{&7F|Q@0c;13cXuHdQpAr~p)vUmfq+w{He{1uAj`OHO^;PShOi(!*@irY_ zWiC~Z@B~hBungoHI^zuulGSKntNEy``doFmv8?uSe4EvDMTp0pc2eW-Zq52{oc9Nf z)3Xtg&OO35=*Yjpas3$rT6Hiwf;6sG}wXtzWdlLj`5Fg@NXgAZ|PrS+2`PinLdKPmG(J?_w z!xvgk4?Z0GMYTFa^7A4l8f0JUv>-&xd~_Hlb*jBt zLh3%cMnyU`@Mp$FEcWf6PnS|K);$k)$=zsvY5^!cMatZSbp}C1?xnJDcl=4dpn0zt zf_O+l82gczf+{lyP>2q0ax9iWwQ1I+_{6OX#u`6+6g2l7{D1319I_UH9FTrebHg%gHO;vm#p#vv+UcKiqtTuaxAC`$6&kB%rVJU;7~+zSn5tg>yrs zrFO~W^0tX+F8}4gkTm%%$Rt!{S-~4^-=S%-=0L0eKqcq*YAf^Ld#al>3nf9(Gv*PG z$cvoS_`~Dv^n9K}+q8skZywL(ELI}sOJAK>fF8-gE~?%XRI;7syVTSHL?c}(s2ewK ztXMt*31R>}bP}757=snm^+6jJ9Mzi}lUs;@Fs0qE7BI^5yu(0pt2cOa&-q2C-|0F0UY6jY}O zL=x|+7sGm9pX7>sw`YA8988TG+9!RjD`JtSD~Ph_ZJ296p_n8ATY$F0zcR2eWA#B~ z`5sXkz|7Pz=%8M97`tYmdS1Fv(k^|vf-2xGsJw2t=tIZTe<(0Aef7eH3lC)U_HQBB z_**p04-|hw{QN!1{wX2JA{D5L#)kWZI|R5TiNN!j0Yya>Z)RrpWiMOI>Li2eENjB= zLR_Slz|Ofs(Fg3V$>mJw;`>BbU|(m?M2#oleSYIo)OPbqhOp8mgAVi zk8YZprf*ncvL+d05U~cziV-&wxk-Fx|GclSZ?h7VVE^azyW$h&@86{`+qnpojZ3jq3H<}pmp`#{-UJUuh*ZlDE{XaFAa+wzP!>5 zcjnTsYy$0aCE?Mn?HH~)XVO`Xc}|k~T}D|kAI)YcHYR2Yx*uc74REa>(b2V?FpQj4 za*!|&upi9~!vmCGiwIIr0-fFmE1TQoQ1_=2#$ZmMcsC1+RYNb&^3Jy`4VvK=1L;6TPQHl;e$vJpOQX+iq%T zG=BI1WnG)<=au8RqIsHmc01DqsL7+JszG-KG$buuEyid2D*Hb8WZ(syJX)I-0Ni!I zmqegT7oJWN1$LIegnE7kn+wLGCh6|J2|Vu39PKL{{8mH`O^8B)XCX83oQ19pprGkX zhPi@(zS9L~DDY6X9vqIoUJ5`0;&3aG}?!GY!WYLEcqyS$^6!nfxaX`M++S2M%t zu3bw^v29@g@ZrPueH=tDjcocgiC%W`-k~Dk>nA2mYx^=m#{xaQ`#gn8X*foDNGJjI z$l;Z0*8i*8nJoWdSXf(E*NkG!Ly^5Uj$)-aerYcrA)%^2zqb^BY864pTXUOuK$h9} z21Vx!t#YgGbN@MDKvGBLNez^)fIMhvUx2=R<#Is~lb$)vNe@q;Q6QRngCwI_6Cp5y zjusBj2KWC~+kEM3|D0<>Lj$mf#1s^&N=hT;)|Au)`1p?wR-O_A=v2170|n1=_4?-k z96T`)H$(1kSlLtpX+|g{F{gifdUuKHme_V0${S>VS*G4N}7 zJcX;*fQgPyjuN8MwMY@rWfbhiXd9-bmH^KIR4XEMv^Dhqt4L;73wensk|Wis|0Mo!2;Vw{?qlEw+5 z4r_uELi0p7xrX^Jh@$1YWqpL?SSc`|U?3+a2MWO=7}0iv1tPLW7%ZY4j-e;UfR_MC zlyRk5fy!fe;C~sE*zRmbB!kfB_{;5|KFxs=`<9%GPmR|vD{ARPsOjImdq+S(aB~O7 zHB{xmGv)!Uhr^L=c>Mv9CEMw10J`B(GCqI2ff}&wl7e=>XkQ=3Q z0JSAOpJ@{?t5!ly*H7Zd1O6rib7Bd5Z@2I$0ZAd~zfPD-gfSJ{27wFku8Jhk0Zx0M zo|fDIdRI2gLqyU0dD`u-qW1$RdP_jKdj7{>MeoU`yHDPR2Tj2s7%x(NX!5`C`Cnx( zF(}Br4*qP_{n^xS<2FruNDO&!uAB1IDV;|X_6*(=2!RLMNhB`TBZl@@lY4O)KnO_3 zi*-taVtSR`o$UWhliRRD!VLl;)Ae^LDMm_4q+CFZ7Xf#~39$PC`$J}eKaZ-0NPAQ?NB0eLUCSb8W`gK)Kmd_A~8Rvrqm~3ztUuLDkeVfflk*p z@V3;mq@+>kD)H3q-2|^_Nq&RqhJi0c!skkWZeWN~l5hp|l7hZpza7Q~+A%mt^!f8= zb#?WE>!1XJz$oJO^Tu65Qo8!-Uq=}84e&*bBJQWJd6sypm-w82PL%BRGcSr@13Qr; z4ecUmHc^Se3Q%!8%w0j?CKv?J5mvfB-ZD=NsH#C3MoUXO#0*8SlO@y!{Xm3RCf^JB zKx$i^1Ee>z+x zC*yjJAy45aM+S%&*&9z1654BTUeLD2zdTzDJ*Ible}uLDKwmP%ubnHmKhk`OJI@?p zNdFD9g$M2Ooy+ZmT!2VXK)N1A?^Hd!PH2ty_VjGu8{&3pwFZM7rF*(~=;hB=Xt2TU-1mPl-DNqVWb5nB!!~uEYyI1^Ai9!WC@AYZr z+HM@=KoNp_BHBKFd@15#588acl<97DhMS5vy(id>QGnJD$3&E*@&edZ`x9FF-JY$q zUoQ5!_J#C4^_j#CAU;Z61Go_$C_z$20|W-?2dt9Uhot>~f%7Duy%~4?-XgL$K@z|v zc?83rL1`es&)<~Lvl`WAUFb=PDknbzKFU4wzo3a}ntifWggwn!?{^71%Ifsd>giqf z+w-KLpop~|@RV)8#Se<6TKFsd-@nEHkXl$60kLx%A7-rE2o&P%(SB6;C_&WGy~(`c z_>q2vjdUD9KO>`YFcs)Ctj54`%cDicmhjYWr-N)2x(t}H(kjbXF8vm5TvX&=^wjj-EUuXw%2ahppb3PS~?SVa5JJQKSn19g!N*!)s3+T^qjz%9b9_ zP~_{>|9sR;YlQu8A8p}F>XaBa-0b|9?=QbUloEyOZ;YxxnoNlY=JDsmtLp%%f?^j3 zO7HXZP}%qk6XWue1)Mq@Kz$LBV6=rT9(=%exWfUldr(Y4BaG4kG#J{Pbv`?5z^0xp zItOQv-a~t@@F|S~k(@w91hfXD%2aZo*m@QV3eRQmkLYA3lpx;WI%1y!l7v>y>nRAC zLU+F|Kwv<$C)073yWmU^{UPLx2wjMQ;P-2;_YyB}2~exlg6b*~@p&=qofx;DeQ_A; zo5hAT&;mky4~fyD*RC1*?U)Nwmc4$>F@`6xezYZA5Fh@VCWY!QoKui4?#}}{8{63# zY1ojHolVXl7%?jCLn{knn!IKu%m37*(Eo2WDJZ?r!yI7M0oiSGa&md}{mj%<%(G`w zgU!Hn=)KK>NV&JO{@fN=UztcWh>PAqY3Qtf;)#WgC}ms-ZYQAcmRLrJmf)71fWORG z{P%Y(0R~YoA5Xn~wB#)idRN}gB>x7IN6LThVvWu+Xa{f}8Tsu5x3RTexOC|;gHV)n zS8?4w``B%X`0nFoaO=o7BPAw{2m4!fpyCIGFq*?8P$t}+8pF5$`qZ{ zEiEn1EHvsseXDBiJ7%*jrJdWce=SSpbT=~ELH3rGR%}ATZ#a{(%F)JVfxeg8Agf~h zcS{y`K=AO%pn!{>%L4EExtRM>7iLYuW!zES5!`bZ>@i!J!GMbNZ{>we z#$vGuTGDfd&O_Td5bc{1b-K{<7*f>@a=_?*fZm3Q&-4ME0(4cpL3*VJ=ci=s?YZrg z^>htc$uQE=(trqQ*+0+ryoY|;YV8CA)2$CTCC6M|DjQdgX&4-$`x$v+qI7$<-C54LqFTWVHuV@ft3oc6Y%Nn z+qXeNF6zFZ<0Lb)w6+ZA^N8AvsqVL^<)wahk`Hnwy?#5<%70Gp4ZkAN*=IoP$b2j8I)@Nqdj65Z3@J&>W(|B zjVmZL0h>VloB(YW&**g!uND^Qr%v^NWMXH1rk5W-8V*Pv z4@rIU@}(S!>uc~n+ut|KD?rQ%JQ+JDXGE-Z8_4@{rA0TjI-zsGb`wPYSocir(udxx zA^?K(!(jLTcg>U8{IQwE?tofR9Zd@s;cDr`jAF_eXVv zQp!VG@oF4I$ zk9}fl^N?|>%KGt`m31+vbP+3609u?MAESYNF>OibeevvBV{Bx%ef-c#VqX<;2RTB*fr*;rXmu-nV>?nFPEuvh%Vtgbc$8>*&sN1M6V z$kNi1uL+y+IzJ||+Z6^f2m>MeJCLD8y%>vlm?&~K5&DAjJy|53?*T6(uQWJ+{ydw_ z2IwfkG-0Gu!4oW1Q2b<9K#!VHz#?KJ-FS%`l3JETiA5(`Peo2H)ts26pzIp!Lm~QR z&*+ygXP1`7;H3&f2N3Z{WN;%>%LRg>M`^C2F+5cuWN-NB*H$If^|`lKfR`6LKJE{` zHF4Vtsc3(oztugb0)LaUR}AV(Rdv1N{|mDu#lTG`t67;!D4O#=U$5nsW_IO-*HF zLOeBwDnLvUCRA}eE?ck!Ml$|Ygrtw^5VfQ_|J9s;vAV5%lVs1SpvbqO?96~1OL9M6 zYS97Ddb)P?^L3WWZrxNka)+SXSENkW2HM-h$$AU!s>fRc`KG3J0@TK}pF~KwGG5nb zNF2h>TRU5I2aBK-n3<845^&>E5CB^Rfr+lJF1!lkH`L=Si<}#@N%%}B50P1F8^EJ( zD9LDzd`e;G`zo%HdTq=$^y8IyL#oe#COW+OLDv%0N0|w;oGYwHkfj3rnCs^)XgFD! zAc6IUtn7LW>f8Pw;55{k>EQ(i^PG?^Wi10iA%@hUxo*1~)`G$UP}05ma8jZh8hBXU z`XD`+9F)u!vje5)8B!rzL~xwOQG&C?l=GZ@cd@$PP99-PJ06>@i=FJ{d?_GNL_JppB@t;*8;S9B(7pC4VVL9 z&IARi6`@-?bkO$ST)%v1;VlY?`lJD6gJz`Khl3EoAt4hY%6eOKIVo$c)NsfFAwa*X z>F7Mnx3oDc%nG`uy>8t6H?RN*hb@JxCzUjdXi&QG;glQ@<}K(`r%ruN7HxlgOs#h| z6|(laEJ?vRF0Msvn$+G|oWeAfa`#2O{n?gVw?xFRDUHM~BY8yqR3!b&i;yuM)2>iW5jI21saVSke5TkRkij*~!JtJ*^nYn_rfMNLl<{8+r~ssWb%6 z=lnk^QfvU7$Hd38kB!v&{s!#@mo5(wC2r6lG&eUlCGw!v>X8ac&T}oa0-BVkX5(o> zLIUtOj*cZj$W2X*=J^6khD1ZGB!7K2jn`!l|_FK$=Dw_*#;$&-nA+X~Gk| z^YKZS*zbbO%m>9LM6O>K$LmsfE`KO!3vyGnZ8o_vwVhfP4=x91Sq>LZi7*~s&rohV z=)Aizw=uB{jW#$0f-x@Dgs2k}m2C6#RwOhTTwTjO1>n^bNO!ms*ceE-*bK#kKTI%o zeo$IFw6X0I=EJE249L|1UFFbhLs13__piX2_w?LsaNGi2R4?{4C1o$zDLx5J$@Muw zsILdybE~0lnNai}`VNeLtoTFzWAX9n7qJgz#m}A%{9JxWBXb5;7JKTk^XHdcR<$*2 zeAu=sX?|H6*~tGg1DFZKO&~Tu`g6t@3Upij*UK%u|2?Sp4a6+)GLlrC9C-B!97Wq+ z`3{{Z2o*-ni*cQ}H#R=g$=6e5M}GGTHNojT91qw7=}IUmeY$7VqZp1t>nTcoC%W$~gbi{e z*Cu#_B+T%SeEB|TGI+Mpmu)a6A09CKZ3otPnOV8o50$e`g+LhsitBj8M%a@%u(Yh` zT!1_U7qvqP&JfdY3~|FDW?@vJ?>6t$tfpCR#6(q_HC+~AC%3Fi9R|Do`*-jBG@P7) z-gAJA%_{er#5y>p1^TqOpTWDZH0>hH30fK&1rmVK%9D9&PWLg{bs!@zPbP?0Y>oDLICEA60AT*ayw-_ulhXN9g9m9K2EIe*4 z+~`q_wmJ{AVY#kw^l3&$5MH;mNt;e7+|}D@(mE~kW>u262Un9$Ci8? z97lX$C=MsMOPrT{jlDW$i^<{O+uQ9J?NWG+0l7KJs+1S{t(EbqWsdZILybF#Y`tHA zG^5;U+_{{S0PSU`@V?EEd~>$j3=tFD{t+lMa=eE7d66liIUlksa-8mn>pK(leQAUE zrJ>DZdVvrNHgvddnrmwlPxKDD#TV@lE``Wgnw?VaLlC(|5D$md$e6CZ+~3{ah8MT! z@cJcdd`QJK5+l?yTOe2Hg9maVLDiH_ffj(zZSPGqJ(|$uJZGKCce{7@Qso5ixdTr5 zG29-TmiT@+EBlZ0f^vbB?C>n4XLq=w);Y!q&|63fS=W30`l3CT^>R-*0nTuN?zDT< zde-6aC?rQshpAiUk$9!5>}qX`xU*?7n+JnZJVt(xGvIx=DG}MmDA}0*fp3Fj@TAfH z8tqb{vscHFz{%}MLUQDzD=j#Uab_AQ9(OcHvJB7v*uiYyvbuuCj%+Agw#W4GB zbOWny*%=Q|uxaeOfICd~1ds4NID*ww9DFcubdav`kc-3p95?nMLt6RjnD=t2c{B$5 zvPf6slAut8BZzwNGrvS!E(Us8>##u>c^Pi3bp1MRWQ?uuY4h3O=i5ZWPVVkV&isj; z7tpf^zDP>Pn_8*_=D+P#_Fa<{12r_9IiWIbDmH^RwTO#|==;$&KZVmQY@-34szBGB9a%UluCrzKwhbFvBj6*Xi5`-`$v5M4m$C!y#9v=+>y9Q z5j)6_9}mnbNsGuyTC1i=HKlup`WH~jEA{vr&qMUwR|6+TsY!INGHekk^Qwyf zam|_(!&O%16T-s5-)kTl0nCPRPu50TMYyHg5Mz)m0rB;U*zp#93B698_4t7Qj5{F8 zoL^p6<*BwBaCUViJ%0Rze?c(+gQFv$073Z>&C1jbJbFYe_3(VOjwMt|P%!xU`2lh) zKZzF3NKbE)&BHYdoCZQ883f6VUO^H(Qh9aVPn13K4v&DEnUzyTekhl;!QtWIi7$WW zI`?60`Q)A7XJ{0dgIi2X4Nh-a`*=7yK6(86G~^0X`_z-9I|0f)VKeu#0<#MWlu9Wz zjZ#aeAKBT`MMqJz=mq3w4^%@_MdvTNFc!x0!h6nudwA)2wd5T7b>E(9F7(hL@YyUK z(DmMxv1?usH_jMV0Su zNZ?ffWQCKR0Qb$vwr8GK(2Q{vv%xQdcn66+f4_k+2}#%PJ!K|H&F+>=lWJt@e8S!`yL@t=0DJiaFRW&;hr4oawNHey81~3vH+sYn0{xRI|Z-FfK%?4|8+SF zRM#>4+BUgPu2*JLS6Yb*SkIlCfCF!PfjEn)DpaRVePGr`UOa$+LU|b(83hGmaVCk* zv_>bWWDr1xysb;G{Gm8+DkG4IpWE6d9&_9}g3Se?hHVD#Tpux?O1+2;tmkMS3b1JE z>$|nT5|^i!9p87n-)pV)to7W_JzUp$o#%PY8Jl||IveUNz$8D|IGopZZy+k^ z3Za^@rW(@&n*R8)cC=yNdSmW{@)R8k+X+o&)5FNgP_`e@VaZ$IvDi>)AX4)pA#FuH z<<37qb#dnLzNxKacRuRutHwNL78rdL!fX&>9QW?+d&=d!uzLfPANtZc(3K$5aK>Gi zg9VNRy6+cxD<#y(*qGa->ZJrn>KbB|qXKFE@qB3Lmi~8RcE-2nS64r;to#%vYN z@fv_mIKfJ}ALpJd5U{n}LF62-KOgb^qH};q>LetxZkyPj4^lg)a70pImCZvebK$IE z9hTSAHScTOyXH`&jtFG4&*ZiAJd&mMK8AC}XKy0Y`hWT|r9EH5t+hbF?;}gGP^FID zT$M{0wZICgWy=ix9js=OTm}kfKi@XyDb(S2a$8B>{YLFruVloO(H@0y_6?K>@G!l- zyzUs47HswQ_C}bGco5QBQ=Y%x1JDo1hNh-w1-ULyrYaxXSAIUMK%lpviFPddeeZH$ z3&1OGFLG4GV4HWnD5U|i@YNt)EMCVf5IBWsnSOz|AlYOMRLlabsA`|NaG_&4Ljd!i zt5}2SEnl&k$W5VFAkW?>gAy|{b2hmadKYj~OlALiVS}Dt4y6`4_5XY!d(+9Z{jG2# zS+klqPL$oI_fT{;cuJZip2 z_(s||O{Hdkp@aRj@L+5yN26BoDhLZTGf7)cc(KbfPN6aa%HSpY1u@?S{@)t5oD%V8 z=%jWP`l>1`UyT#j+TSG!Wx-j^0#Zlm@4*t@WVmsoiqB7ZO^p+;=~_2+o=fZ^_0Qf} zs1$r2uf>UI#Xx#ac=oKHeX4hsMzf5Zz0(18n1G;Qf0Ev{Em0QK;z(!M`fU3%UAA35 zOVM$Q55A0@t1#USwP_%`zG)??W{LU6m`TW(Y^`^n$jv#cZ#Do!CgUi@)5~v;+i!M- z>nPyI?by<+O4KTkuS|eiVWNxi8t~U0u({V^V-W zJbU&GIXy}g0l85`~YeO}pSd621r(xzejAPo5;a`P5Ic za>Iws!Og7?8={ly;>DorSV#Hrmx(P=yMEyxgB|R=ym1KbcGKn#NiSV`6Y$ zYvDAI?^L^Nt2jUZ(q${;Hk+6z|HGoCd(Lga6Lq*T2I&6vytk;*=}$z3goY6*B0aA? z`^*392d`uNTM9H?9b?VrGBj{Ctd$gz+#uPrDqbl$YsH9kIe zDW#zMh=)i0j&Z*S7N1RrRlwdsVEyLS_I|KvAe0f!%|Mj_H+^p^^L6_AVL>7#aePJ1 zT_n%YURX#X&36MzC%YB|OMf7HFaqf*BB!3buU~ygXh$n}`fpL#f!^c1nEF2`xV~hn z0s{#O=**RhJ5v%DtpEAeq1k^@BPMyLB1C)o`(+M?eP$bbsHMk8rEm@`y2yBXDPy4C zvLVJQI4}~N2k^Fz9)^sxE*k|6qAl!< z+#3VeGn>;cZjhuu`(#6FKqYDZ=5<;e0mA+;=&-Z1!-ZUw-+>UNd!P_C3#9Ol{wFMN zP_9HolqK8xe9Z&=m-Cr`8L)r9w;MF$T6?}DYsbAzxWBlpUPz{1w5Jg)8`N5#hDm(y z?DRyDK)KoVy2;j5UUJZW(;h2X)n~A8kf7kbJ|TmyRS0rvYL`qi=A{ z^OIfa@eObaDhuxcvcoEF%cK? z{O3UYDDz&7YAm6TM%b4V1DhHGVEpmp$6MR2R*Mgo3Ke)k>~XJ1VRMm*$LaTi@Aw4; zYkgy?l3&^n5SNTlrNF(yoW>i-FS8VEOT`E!I>}+!__%nG|8de^e{YSmFuZf%o$@y; z_Z#}bm$CV0>dZb6!_AusE>5#Z$ve6h{0iG(YVqaI1OdL8X`?)+X}OdX7V)WneRgRK z3ieTgu_vOcTN6({ldvM)oBc7DA*`fg5&K?Y{*L1cTqhh?2ep2G?WY**aV?r$jCbRi z^n=4R6G-2*XJ32ZGjtF%-*>ij#)#8A3&m8_e8=XpxC)N)Z4A_&7f7*K)D^luPtVt+ zP5130aEUs@q$sMh-@Kq5Lad`=-b_TOL7#(}Cik|-?08+$p9IQQLs$S205CNo^PFxB+;Wa7@}N)ZT< zjFq3BTmK*bHT-sd{x+X0>UsG|B--pS(OjIIoEpE^)Zefs$6ZZd%RrfPoc$(c(OHC> zISnrVfY`I9HZ|onvSp|BE5`TAkf^g`BzY5f5>&5XL|dGzG1|EC89Yu$;z_-;L%^hs z=|46;-uCgMOf5Z4*J&AR>=bkb`+VITGQt|f znlz(7!!Ic%b($p_@;wBP0gCazmq82TQ76~qzgWm)iX{5%gezV%C7$JyWc)oyX?!6G z1SXuh0I_*Z+L{B6WWT_nL>)c03y9m=1Afn!XZpr&D8!*+AENuU=}}A^W&~^uWuK8H zd4ErsB~iNv$qonbrpGBn-BC~MA| z^F#BWlW(ntS;BXunAay!m5=$Bxk9ejZTdz}ZDxxGAj<_Vi*aOlcwL6ocZtbiHe&)? zLg@&+KzBf`I26X|H!Ok6+Z<~>4Rlv^lu3fcsa&q2lDs9|66GZ-!WAfBEX>W{RzlYr zbow+QP!5qs5qdzQSbt{p){9YLD%;xG`Ba-IZoFMvW|o72k6+tNz7FdPlzk38g*ded z;Lg!iU)faU@Ez8%Q_Pxs2b0dC8fwl_O#ruW#m7$6 zYW|3si8+W9fCk$anCs|knj^P?(anv^#!%?saECM>=I*-=N@1WDs6HobU&i|Mm3QHs ziEcs-jwrEr5L8JJIr3Y@O5V)O%R+IgqLLDPCi@996TckQftTuw7PX3mPL4NECXIwU z00zOIzMBq=bZSnfWol_f-&v_okZja;!8myGSJC7FpXHj2WaG>{^g2S*@SeH`h zdiuItTYjUbgo`d%r%!ZCif6YBUPN&w<6$u)EhS9eudr)yd+@Be4RU^(e{zP3iIO$p z+XGVG{itv*oQ9aqN!PtFN+1Jd(3Q17QW>F}!EbCjhbweD1m+ydYd8z!?PxF`DqzmfMn^3#F=~v!;V7 z6L*?fBI}@w@@-q`D0s=kJn*~U&;!(3V+z((#itnTx7#EuVQR7VA#ooPv{BeZgY%gY zx+J9S>8?lK9;kZY$ww<606W&ZRCP3^T%*9x0sRKdF^{n~R?qhwpv%*PSLu(*tRkz zlnuw(0~fOoMV;EsONJ{Jt#AT4FL+vaH}iA7iGptr2z+>y)s>$C1C3n7a z$S*lx<@g0d4pN2BK$im653HfeO4@FKH4Ug9LF>N$&-c4JyY3kmJ>Xg!RKxYvtE@z4 zc@k32`S$O=z zP)|dfuMgMSZC~(oiOdsIOyaky34J-3n!N0%GMUk}IW0Zgn)F?opc0FH!USm`w@ zsB)5ftW3g%sL{J59P&*{DLP@=Q2|vq6`Zr_&5tU-R`b#clT(<4-fpgLySk9dbK#&Ct0#v9czIhhG zX=5*>=lnj2+`PQs2?}?zakEz^tj3UEinTd$^+K3s!k9DTe7Rb?nwu$eAI2Fm3)bvQ1Szc}a##ar*256o+~TD(Pu!^Xhf_ zozlGT8sS|P_2hn2=Y{~O*;q_7QCVRA{6uBxJZ~ybm1o*>johpQH8&0J7H7~G9d4s+ zk{8Jq|DBn42!by=hnII>)!8k)#wgH%z;jK4vlk$cPU^<|oOtbSIK6A~`_6`4{piZT z1vj~$K{4sUenbhi1Ll?eIt=3RKWK~3o|gX};+ zlihdT@9?qIr#-@I+YX#1kK2nSskmY1}l0UuYmC%IV_x(@OhD);4!L`fRSi1(4NEb#?wJ&gAtg*@9}OpW_)r9x|wu zZOE5EWYv6iFjLhiMmZEi`D2b8$JeNlqq$ksus-#<;%V>Qckfm5=(hCJDsxzn1D3J> zSewL{Nuc4(L1Jl5yVKw@Ah!;77pU`3_;+6`2AnyvJ>o0=K^)ZIbZ^h`_bH1P(|&Ne z;_T76UuFE7y$sv08Xr9P4UzI_istxd^zaztIbnEWdekjVXblh{fezdEUF5@IP-w$WfNBe2pu{Zu~GVFA=(d%R=?8;|HV7_k(c zJIx=X$ zClv|HGru9rvoovb6-zWQ7s3F=zwEU8FBCwWntwca*WD#sJHK;?r7Kp`-w(bqW8auI z&!jKw1gYRL979)uiFSdC6@ZbES&x1DN6ouJMp_GQ3WU4VkmNDunR~cTxe^}!q=K_Wao+rfa z`~e!mNA)2&mYHkDQ||6A3&9rqBP&Oedgb=OE{Xr}5LWGE|x}^YgD&c@FeYHNLJPmhsCTw4Mru{q`p!jMUHY z4Dw!ARC_z4lF~*EJ{KPHOz56!I$z#A>Rk*n^=?D{$j;OBD{x(A6(PcQ!ZONOT>*+7 zKTYkj8Cb<=E)^Ev3;FrUUiiDB!oX@0tGTLn`Tfx`*kJ^4kev1%RtN4^+}GLUOfp{Lwp1Rl@GE zOG^|**BOlLlq@3_wqN0Z~Tj`KYYZa7qRkDEiW8Zv63U}+nS9k|ad{@s7yp>0vnZoW>l*oTAxq;%EeqvS^iK7d{Ti#~&| z(%AAtKb4!l`3zOUqsU!%cC&0Dtr%~&7mr_CQR^|_&ua8NK5dl^yn|!1gz^?Dt7vN4 zs`uZ%*OZ{w)zu*TbC7|{B8_1NVLq7$=Z#;++ZuSjxN zyv3I^pp}wcbN}!@;-_PgjDI}0D(AbQ9!)`MX65k*2Eo~=2;}0Tk*Y(^uWuO~imn$b zIXsi9n&-`=nt`oZqr&0^TW+|U zFF(cOjk*Xja23~!<1+uH&cYBJA`1cviaW8yXp)%!AfJ15MGP)i`eD==O0<`w9G{ z$vmUVXj7aj)xmEt!`7{+Lx@Z@1^!+iGrBLuHmtQ0tFk)A@2e=9o@tEpbTWq8>pDF> zV;OFzs+x6C$cF}*P|XRJISB95dFX;nF$RG_%WV9|2hqM}5&hc2>HW$+lTwPNB0n1T z#qF(i_Kf#@)ve3)1aL1B@ApF9wkEYM$@5X$OpWgc8`rfIj>=H4JjV=o>tB$lT*{hd zTV!+H;cf%WSW<01q}8lfANL+mfr%C@{(%b zHFQx}bz<>i;p@^AR?I%}4!~Skt38{yptIf@j5*fxU0pn&WL>gbOEH7t9$YK9xw*$t9BJ?^07HpG`Rff3SLmGeFP2dcm^8iu%#4Vlq5c8($K-o2>z4HQ z_g`P9UOJenl*R&EwH0b=1%NQ$3t=cDu83t`4f&X#kn8YiYj`53nW*K7#1UuOB4|Er z0IkoZ8qMH`t95DU8?zs<&7(Irt-6uT;)oqX^6yeTi;WAc+(|SK^nPHJ^?|H)z=oUL zeGQ~jP6_Bokpj~qOz&;WrCu79IY>($h64(bL%iPnh)=9`fJxy~9ZgK2Et`-o|ZJg*eMSr!+6gz4N75 zX-Lflv!2L#FVbc77bK{#uXHK2Kb;yFRI>w{;J@z09$N$m+GeFia^_}P^G2lCL*ui~ z>&GV?h9fKoB)F3o zJ-B`0#t#I;!^au)a1!H8J>bWCLVpIgxiN>#`}KSJ9N-<1J7m}zVvMc<1pX2P5>)@T$#mn~Zs;%h8=hNeNBRE-&97k+XNJVCl2=I%~#!BN#~@f1DA zHxp%4Q*?gx_siC{gnEorufKWJL~~?U&!)QT#oxtD!}lH?TgmilCa7Hp3L+3LaJgJH zjJ;ftE8>Q8rAMxR@u5o7GbyT0d-v_zd~Nk2I0!dyZ<&EpH%KuHFI&ERKXMS1(}%U^ zy5(kJE^lNX}V_}9yhcN;(ls_VFkm*Yor2O{Hv z|GN~c{%`X(jAq*ZB5y;a0O~7nBMY5`?)W%_H)OU+j(M>(B5yl$#`Fc|O;88n=J=n9 zTRKqhx{Ima=y5?gkcOajOfsN`@Rd)}?u)Eg_WD9dumtmZj@evqZ$-GQ@jt}NVydT5uJ@|MksdZFmS!=7Mn zA8CV{#b4>!23=jVK^TR)8UWn;@-9oNQ!s#c_5uCR*&F*1Vnx8-!2Y3tdj+IDGjBG1 zd)Tl}UCri5UDw;b;MzS8?-XonK&>Of)^2OlmU%Xu)1%`mO@Hg6>@5c4lcPXscAH_n zcI?}(yLIbp#2;Pk(fz#n4+{z&-f=;@LLX!wavX-r9r7GmLDLylz9<#>nX*Iqd2q-k zJmx;E0KkU?@ID*@CP-FtmR3>)h(AzBsN6cQiD%|ZSI`N=j{)zeMSLgTG~Pd);HMbZAaW2uM<2hxGtMdtq{Z8!TV63R!~WppTx&)BVE^teG=gPz4VeKGOf(wPP}`y7v6$ntyy-!aIE)r51bs`^?FNdCQ}a`&u-V#cNR`;DxOjLb z-Zf8@^Los=Jqh;}vtJc>fM`Sap+-mU)F}UZzm+)bKJxmr%+FfshywW-J9nS3>+UCK z<|0#>tNG?2l5oO3?SqJ*AWGlp1h?Do)Y9&|=;rqwaM=N65zWU8kvxyC>`VXQTb?I0 zy*jnEX?T25=*mMhpK))mKt6>v{va^TzW%Vf3T5jI^ zUmRR%Tlf&?Z&I)&`zBW5Z+1Yh`j(KN>@VifKj4R$0sq~RZKnl4U$7OSm;q)%EcUP~ zS{f-(9?j%WeM>}X53;=lTDzj07tzD#Paj*0r0LjKaz2Cl4PMHf&o~Sr;wChNgiQ{X ze_h`|;nk)O@H#rzxX;>}FnXbO0Aj8e+#P)%tRCs*M70Ke5*zUIQ5K6+anrt2v?j%| z7GWq_q+%o7tV!AQphWHfJGdj|XfX3@;IqztbyV)F8nA4^1TMK+L%0;&`HVT}3sSU; zweFXcr%CINd~IBlbc9x&6-93v?ua==SoSnSM&8pwRED{8&k>z&Jfl?H78_e=Cp^+( z0@6B!atW~TD+SfyQ~CK>CQ|M~`|N$hqZsi)s#f~i!ODfy0k(5TKOih@U|X8ISw;sD z_?w;FLR&BFZhl0lp6{W?bUSnVjw`57*4`Vz0va6|Nm~@bFCefUr7IWzn&VEvlYL?S zgwQt4yIvvaon1bI=-0aw(0YKBlc-`7^UpD=6}FFKEJUbXvJc08ycnQ? zvW$uT6I(9&Xup7{tgML_$=gCehP9~idcjq4@X#5jaYS6nDbLs6s$mr#k9O(+%jw-s z19WS}I5@5arQfUj9PkSQcOJY!D3}xA)g^GVYg{df(zvwI1CpTbVCGSZj%FHnMiKMP zs@MPQ&^QdOMB)9g)FQKl7NUqM`;+TKf-b5vjlwGM5ifLqE9Nw$48g2pjY5JB?2t^# zw2Ymxd0M8KpMLQ1lybnw)4Oe@#n>C~Aq#=zW4Tr49wx4PC9X^=jfX=L4q#0#c)Gt< zAgo_;EAx%N$6GaU8QOFZRIWJWcEpAog|zV)+H};N<4TpTexYDYH}@9V^_vxC00Pi+ zoEl3OF9xjn)+l213#&epWo`*sr92ZkiA!=LN9um_^R(>89krwisc?>Tt1LfY_e6rJ zlr(6Y?`n^L3dQGc=PmMd(<0Q#%2BvR*(2Yvtm#UwL@eES%~7;zJv+KECi!L6&-mSw z#Q{xcEQAxKFC%Ss>}KMQZ+iY(kKv37N2iraloCCa?HpYoXCxdv}c4~aYf3$(Tr4cEWX#GtAvHjMI)+#9|+`*9^a{M~nboVF;uCU&~u*vNq_b9hr zt@VHrA*M%aPOzP-O#M7^ln?Z&ta7-tuR@C^D?PJanf#lT^>lsz+ZpcHIhl_*^;uY1 zYhpH)O?Ha~aEJ3^s`S2}|Lk!eT(AM>3>%*eWhIQBf?iRXsXBwkTw&Tv zLJ{>gcx1qFt*UC&KR-b&=zs9N@Bxa)A;KVHSm>w2y8rAOTmR$e{p(C~%XREFXbij5 zmF%7L3~f!6p@>o*M|Ny>C6UAXkn!fL)}J@6dbB|SnAR?r0DBr3G>Cua7!Sa3>)SI& zwcRf%iJIHE@MEE|NKX4 z8E+&H3!ds`OXc#p*~o#0mJm;5y_8Tp(uKvJ^)^O>B^){lm~yk+Ge;RDi zvIL&s0uCVEXB7FIMP$+nGtK9&PtGw`A-&uI(>CZ2(d}jM@SHr>SU_Au<~NQZZc<1B5Qu?K ziJU-1<_-g#8q<-df?lH(&qw(zsxj&;|`ev9Uiy?HC@M)YtIg=l=1=XL3jQJ?RS z+&+(p=hXS-n(i9r?B;zP?@I{nMpePxv1*)szi=@sWO#%_SEO2YEGEbXiDDn-^cS)J9Qx&ViykmOWEZRLYkOMT z*FKfc7@g*Mf}}<6-cos{+1zI-8iW?`G#TvR5Hp5S=gM@B#G%Fr2?@mqlx+Te`+`Pz@*`~Z%&{eE z&Nu>%3=Of7^7&JZN%ies&IT=VcAsa$F;7NrP|Gf(4+mWPwr|}~U-~MCS@WPp;fy;1KL_kY=!C-lZ)_#;X^jzl z){MchT~gZ>|L^ZqX(X3Q{% zUX#iQzN~L;9bM(vCa6MVzqzoy-bm*iWrz0AN5NyNo33()v(1i#V}1AYii(g67sQoa zwn0)`bc?L3QnmU-RFJqQ8jgbrkt;Q(h?Y+|48Msn7#)Gk8%R6hwaCKEd<5l$HEZg7 znMBIU$fYXJ$pbP2+dz6kdmLVF?z}zGUewPy3`2j0_I0Lsg#f=ikRKw4&$ie3hL7GR zmv$*}-yRtsYUZ6YCy|Yli_4`h_0ks52QnhL^BHE*sk^>|*+x2>VQU8yP>?v@vMJ+d z986glv{%@g_t(atc}5U!Y4DBb9*y?uV%p1;sXdE@b^po+y}oZznJ!pDtybMi!-kB_ zLBB~YzI8w?JIZzv?gUIFqSQ;7u3L4eY) zCr+^j%yNK-7q6*8MX>e>h$4A!=;8fXa>I7FAI3#>de6Y_&MV=ALRe`U!y`3WZIsFJ&n5&#b+Wg z&G}*78D#OoQs(K>Wy?C++go1*8nH{ynI=v#tt?I3D=uHP^O5%g+h!LNmF+Wv1=ocp zE3VL!7h>tTxaQ2Dr!9)H){eBzO3%p4PF-P{ro=Erw$`N|xlqG((#S#l?~mWENK!FY z;!+GXXxYOs&66tLtCF`=Zskfyo9{Nbj4`tD2a-$kqn)TC)J;^MVHQ)Kzb~-jKD0pS zpbfLztH?7}EjONPqIBLaIsR#qe#!ybiq=T2k$+wK<}YO3Xgb5h$cPXzGCJCMI570% z?r8hiJ;L7n-nUQ#s%?Y8F7mYACA9bdx~&C)ijXL61T7F24-)8n%@OrsTlC6{7Je!=I+Aisjc>cD%7 zY!Uz0*S%&hxq|qjESO)gT6j3(gnn-)Z@`x+-3F<0F^vCE#)FgAn#TZL;KKyM#%3&P zz<;vx@}G1P*4!@R)Oia8kQFZC!;@eW7r-p*_DpM(oTBIm6Yxxz(hsz z8JW~^in4@gCZu@3&$#5o)Y9w4%1b^y_2=RjGK^8>^<*ZJW6<;;e#w|9ao2DKlu6Rh zz?Qn#Fyth$=&(?2+%Y-~vk9!g3~kz6 zEYOxg(CNR3;d)yZO{Uv1F>2cw zY`&A^xuo-n?^s^^MJ(WSE3+W3XUB5vQNLJ7!g2>Vq>*tDRh z)3!ar&4KfG68`%OS_WSVVyFQM{VbzDBNsBwUh{gpQ!SQDoBE$iY>!!AC0RSnAKOCQ z>Mc^1r$EQl30Fw^^RX?mCdB6)y#D-D^qzKe`=8FXccrUH7LRA|uX6>~5{{S|8)?7o zP9d(*_RLD$8sAW|$0BpuG`!bt>t7dUmpY5RA9Q83LV}8*l-Gq{u$HTBX)QgG?%)B8 z8X5%|87eTRG4|C8e|~qjb?__Gx))t-P+$7%0986lM7w3e{e883Hr-+Zp(!q!} z*~MwY@aOK{m??3KsaKGJdh{^Gz%~3{Emx@`_wC^0OVQ)I8M}Ig!T@EXk`3p~XHt0u za=Ztf*Wb*i0#h6_EP2t_xvmS(kYsrR%9FQm-@>i?btVXy=%~IL02S=UAS%qv&c4Vi z`?#wR2w?Ic;KQ{LB#_CL2SV)a?Xf1tl1bYb(L6RF+1Mi?T(oEr1T#Yv-=jkAYWMpEJ#j9trC!_QPLa!!! zvDMU_1;hOcA^7Wio1jLGgGE>i8^bUh{z;_oc6S?4B}MZTbh$&@h0fnEWGSHIJ-U|& zP}Pq1JX^)u3u8$9$|a~x09}!%qn1QkL0sWb4f^33DnU9kJ>167Ie!5+kR(RDde;XY z1STdg&o5Mh8Br|?!JOoQE6Gkgr~{6Pk$icHAx#s5Q*_EMclRI2MZmC!%W;>qajob- zXUcm$nR?+POCl}vfJ6@bB0#84Py&r9a$6fZP?v^wMluX@5#OYga72+xrwu{i2UKuO zQA}88iPJho?qT_XC5i$;GiC3!;%Xc3uSanm^>O?jiE3M& zT7bjrABUqcwUx)Z%y@=oVz%0?94&GW2@C67a$XNI1yu0r`i7TF_Zkz=u_IwpJR0g) z<^8YE^D2f7e0^u7F4U~UR7mtKFW$*jBzp`V@YvjUhJ#*9Aon;#NxXB3hxM#_(p^}^ z)edIsG8IqC%K>B^N59QRhGAQimKRgE!ajLGq!_I{uqZb$>; z=h!4k=dgZBt7Q$PLH7Pqmv-f+x3I?4RI1?`Cc2c8EY#tmzxe^47~)BE4yra9HK-*A zuXyaCM4Gzr5avv5Y%I!On9(VCM4*aK&*`VlWQKJcaD5VK4)KUcq%2|^!MY(jYGW56 zb`b0|Q@aeQcWUz>wG0x!oK*aUSgNFeH^2OawhEpwH4+%Zzmg1eO*Uj|+nx9?5ta!E literal 0 HcmV?d00001 From 49990b764437b4e5bb225bdec0327ae1dda970a4 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sun, 12 Jan 2025 14:49:51 +0530 Subject: [PATCH 05/12] Update chapter08.adoc --- chatper08/chapter08.adoc | 183 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 175 insertions(+), 8 deletions(-) diff --git a/chatper08/chapter08.adoc b/chatper08/chapter08.adoc index dd1c4f9d..2c1f8182 100644 --- a/chatper08/chapter08.adoc +++ b/chatper08/chapter08.adoc @@ -48,7 +48,7 @@ A fallback provides a default response if an operation fails. It ensures the sys A circuit breaker stops an application from making too many unsuccessful requests to another system. If the number of failures exceeds a threshold, the circuit breaker will "open," causing all subsequent requests to fail immediately. After a configured delay, the circuit breaker will "half-open" and allow limited requests. If those requests succeed, the circuit breaker will "close" and let all requests go through. -==== Example of Circuit Breaker States +image::../images/figure8-1.png[Figure 8-1: Circuit Breaker states, width=600, align="center"] For example, a circuit breaker can be applied to calls to an external inventory service in the Product Catalog Microservice. If the inventory service starts failing or becomes unresponsive, the circuit breaker will "open," preventing repeated requests and reducing load. After a configured delay, the circuit breaker will "half-open" to test the availability of the inventory service with a few requests. If those succeed, the circuit breaker will "close," resuming normal operations. @@ -96,8 +96,6 @@ The MicroProfile Fault Tolerance Annotations provide a declarative way to implem Retries are a fundamental fault tolerance strategy for managing transient failures such as temporary network outages or intermittent service unavailability. The `@Retry` annotation in the MicroProfile Fault Tolerance API provides a simple and effective way to implement retry policies. By customizing parameters such as the number of retries, delay between attempts, and conditions for retries, you can ensure your application responds to failures gracefully and minimizes downtime. -==== Example: Retry Implementation in PaymentService - Below is an example of applying the `@Retry` annotation in a `processPayment` method within a `PaymentService` class of the MicroProfile e-commerce project: [source,java] @@ -128,6 +126,10 @@ public class PaymentService { return Response.ok("{\"status\":\"success\"}", MediaType.APPLICATION_JSON).build(); } } +---- + +[source,java] +---- class PaymentDetails { private double amount; @@ -142,8 +144,39 @@ class PaymentDetails { } ---- +[source,java] +---- + +package io.microprofile.tutorial.store.payment.exception; + +public class PaymentProcessingException extends Exception { + public PaymentProcessingException(String message) { + super(message); + } +} + +---- + + +[source,java] +---- +package io.microprofile.tutorial.store.payment.exception; + +class CriticalPaymentException extends Exception { + public CriticalPaymentException(String message) { + super(message); + } +} +---- + +In this example, the `processPayment` method attempts to process a payment. If a transient failure occurs (e.g., `PaymentProcessingException`), the method retries up to three times (`maxRetries = 3`), there is a delay of 2000 milliseconds between retries (`delay = 2000`), with a random variation of up to 500 milliseconds is added to the delay (`jitter = 500`) to avoid synchronized retries (e.g. thundering herd problem). +The retries are attempted only for the exception `PaymentProcessingException` (`retryOn = PaymentProcessingException.class`) and are aborted if a `CriticalPaymentException` is encountered (`abortOn = CriticalPaymentException.class`). + + ==== Understanding the `@Retry` Parameters +A retry policy specifies the conditions under which an operation should be retried. The key attributes of the `@Retry` annotation include: + |=== | Parameter | Description | `maxRetries` | Specifies the maximum number of retries. @@ -195,7 +228,7 @@ A circuit breaker is a critical fault tolerance mechanism that protects a system | `failOn` | Specifies the exception(s) considered failures contributing to the failure ratio. |=== -==== Example: Circuit Breaker Implementation +Below is an example of configuring a circuit breaker for a service method using the `@CircuitBreaker` annotation: [source,java] ---- @@ -215,10 +248,7 @@ public String getProduct(Long id) { } ---- -In this example: -- The circuit breaker opens if 50% of requests fail (`failureRatio = 0.5`) after at least 10 requests (`requestVolumeThreshold = 10`). -- It remains open for 5 seconds (`delay = 5000`) and transitions to the "half-open" state to test recovery. -- Two consecutive successful requests (`successThreshold = 2`) in the "half-open" state close the circuit breaker. +In the above code, the circuit breaker opens if 50% of requests fail (`failureRatio = 0.5`) after at least 10 requests (`requestVolumeThreshold = 10`). It remains open for 5 seconds (`delay = 5000`) and transitions to the "half-open" state to test recovery. Two consecutive successful requests (`successThreshold = 2`) in the "half-open" state close the circuit breaker. ==== Best Practices for Circuit Breaker @@ -226,6 +256,143 @@ In this example: - **Monitor Metrics:** Use MicroProfile Metrics to monitor circuit breaker state transitions. - **Combine with Other Strategies:** Use circuit breakers alongside retries and timeouts for a robust fault tolerance setup. +=== Setting Timeouts + +Timeouts are an essential fault tolerance strategy to prevent long-running operations from consuming resources indefinitely. Slow or unresponsive services can degrade overall system performance and reliability in a microservices architecture. The `@Timeout` annotation provided by MicroProfile Fault Tolerance allows you to define a maximum duration for a method to complete, ensuring that system resources remain available for other tasks. + +==== Why Use Timeouts? + +In distributed systems, slow responses from downstream services can cascade through the system, leading to resource contention and degraded performance. Timeouts allow you to: +- Abort operations that exceed acceptable time limits. +- Free system resources for other operations. +- Trigger alternative strategies, such as fallbacks, to maintain functionality. + +[source,java] +---- +import org.eclipse.microprofile.faulttolerance.Timeout; + +public class PaymentService { + + @Timeout(1000) + public String processPayment(PaymentDetails paymentDetails) { + // Simulate a long-running process + try { + Thread.sleep(1500); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Processing interrupted"); + } + return "Payment processed successfully."; + } +} +---- + +In this example: +- The `@Timeout(1000)` annotation specifies that the `processPayment` method must complete within 1000 milliseconds (1 second). +- If the execution exceeds this time, a `TimeoutException` will be thrown, and the process will terminate. + +==== Best Practices for Timeouts + +- **Align Timeouts with SLAs:** Ensure timeout values align with service-level agreements and user expectations. +- **Monitor Performance:** Use MicroProfile Metrics to monitor execution times and identify operations requiring optimized timeout values. +- **Combine with Fallbacks:** Always pair timeouts with fallback logic to provide a reliable response in case of delays. +- **Avoid Overly Short Timeouts:** Overly aggressive timeout settings may cause unnecessary failures, particularly in high-latency environments. + +=== Implementing Fallbacks + +Fallbacks provide a default response when an operation fails. They ensure the system continues to function, even if the primary operation cannot complete successfully. The `@Fallback` annotation in MicroProfile Fault Tolerance allows developers to define fallback logic for a method, ensuring graceful degradation. + +==== Why Use Fallbacks? + +Fallbacks help to: +- Maintain system availability during failures. +- Provide a meaningful response to users instead of complete failure. +- Improve user experience by minimizing disruptions. + + +[source,java] +---- +import org.eclipse.microprofile.faulttolerance.Fallback; +import jakarta.ws.rs.core.Response; + +public class PaymentService { + + @Fallback(fallbackMethod = "fallbackProcessPayment") + public Response processPayment(PaymentDetails paymentDetails) { + // Simulate a failure + throw new RuntimeException("Service Unavailable"); + } + + public Response fallbackProcessPayment(PaymentDetails paymentDetails) { + return Response.ok("{\"status\":\"failed\", \"message\":\"Payment service is currently unavailable.\"}").build(); + } +} +---- + +In this example: +- The `@Fallback` annotation specifies that if the `processPayment` method fails, the `fallbackProcessPayment` method will be executed. +- The fallback method provides a meaningful response, ensuring the user is informed of the service unavailability. + +==== Using Fallback Handlers + +A fallback handler class can implement the `FallbackHandler` interface, allowing for reusable fallback logic across multiple methods. + +[source,java] +---- +import org.eclipse.microprofile.faulttolerance.Fallback; +import org.eclipse.microprofile.faulttolerance.FallbackHandler; +import org.eclipse.microprofile.faulttolerance.ExecutionContext; + +@Fallback(FallbackHandlerImpl.class) +public class ProductService { + + public String fetchProductDetails(Long productId) { + throw new RuntimeException("Service Unavailable"); + } +} + +public class FallbackHandlerImpl implements FallbackHandler { + @Override + public String handle(ExecutionContext context) { + return "Fallback response for product details."; + } +} +---- + +==== Combining Fallbacks with Other Fault Tolerance Strategies + +Fallback logic can be combined with other fault tolerance mechanisms to create a robust strategy: +- **Timeout with Fallback:** Ensure operations terminate within a specific time and provide a fallback if they fail. + +Example: + +[source,java] +---- +public class ProductService { + + @Timeout(2000) + @Fallback(fallbackMethod = "defaultData") + public String fetchProductDetails(Long productId) { + // Simulated long-running operation + Thread.sleep(3000); + return "Actual data"; + } + + public String defaultData() { + return "Fallback data due to timeout."; + } + // ... +} +---- + +==== Best Practices for Fallbacks + +- **Keep Fallbacks Lightweight:** Ensure fallback logic is simple and reliable, avoiding dependencies on other potentially failing services. +- **Provide Meaningful Responses:** The fallback response should maintain a reasonable user experience, even if it cannot replicate full functionality. +- **Monitor Fallback Usage:** Use metrics to track the frequency of fallback execution, which can indicate service health and the need for improvements. +- **Plan for Degraded Functionality:** Ensure the fallback behavior aligns with business priorities and provides the most critical features. + + === Combining Fault Tolerance Strategies Combining fault tolerance strategies, such as `@Timeout`, `@Fallback`, `@CircuitBreaker`, and `@Retry`, ensures resilience and efficient resource usage. Externalize configurations with MicroProfile Config for flexibility across environments. From 4fe673851929a7b69b9f66de959eed828b1f92b2 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sun, 12 Jan 2025 19:47:36 +0530 Subject: [PATCH 06/12] Update chapter08.adoc --- chatper08/chapter08.adoc | 182 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 171 insertions(+), 11 deletions(-) diff --git a/chatper08/chapter08.adoc b/chatper08/chapter08.adoc index 2c1f8182..4bbb7aaa 100644 --- a/chatper08/chapter08.adoc +++ b/chatper08/chapter08.adoc @@ -309,7 +309,6 @@ Fallbacks help to: - Provide a meaningful response to users instead of complete failure. - Improve user experience by minimizing disruptions. - [source,java] ---- import org.eclipse.microprofile.faulttolerance.Fallback; @@ -368,23 +367,44 @@ Example: [source,java] ---- + +import org.eclipse.microprofile.faulttolerance.Fallback; +import org.eclipse.microprofile.faulttolerance.Timeout; + +import jakarta.enterprise.context.ApplicationScoped; + +@ApplicationScoped public class ProductService { - @Timeout(2000) - @Fallback(fallbackMethod = "defaultData") - public String fetchProductDetails(Long productId) { - // Simulated long-running operation - Thread.sleep(3000); - return "Actual data"; + @Inject + private ProductRepository productRepository; // Access to the database + + @Inject + private ProductCache productCache; // Cache mechanism + + /** + * Retrieves a list of products. If the operation takes longer than 2 seconds, + * fallback to cached data. + */ + @Timeout(2000) // Set timeout to 2 seconds + @Fallback(fallbackMethod = "getProductsFromCache") // Fallback method + public List getProducts() { + // Simulating database call + return productRepository.findAllProducts(); } - public String defaultData() { - return "Fallback data due to timeout."; + /** + * Fallback method to retrieve products from the cache. + */ + public List getProductsFromCache() { + System.out.println("Fetching products from cache..."); + return productCache.getCachedProducts(); } - // ... } ---- +This example demonstrates the use of MicroProfile Fault Tolerance annotations `@Timeout` and `@Fallback` to enhance the resilience of a ProductService. The `@Timeout` annotation ensures that the `getProducts()` method, which fetches product data from a database, completes within a specified duration (2 seconds in this case). If the method exceeds this time or an exception occurs, the `@Fallback` annotation directs the application to invoke the `getProductsFromCache()` method, which retrieves data from a cache. This approach ensures consistent service availability and a seamless user experience, even during database delays or failures​. + ==== Best Practices for Fallbacks - **Keep Fallbacks Lightweight:** Ensure fallback logic is simple and reliable, avoiding dependencies on other potentially failing services. @@ -392,11 +412,151 @@ public class ProductService { - **Monitor Fallback Usage:** Use metrics to track the frequency of fallback execution, which can indicate service health and the need for improvements. - **Plan for Degraded Functionality:** Ensure the fallback behavior aligns with business priorities and provides the most critical features. - === Combining Fault Tolerance Strategies Combining fault tolerance strategies, such as `@Timeout`, `@Fallback`, `@CircuitBreaker`, and `@Retry`, ensures resilience and efficient resource usage. Externalize configurations with MicroProfile Config for flexibility across environments. +=== Isolating Resources for Fault Tolerance + +Resource isolation is a key principle in building resilient microservices. By isolating resources, you prevent failures in one part of the system from spreading and affecting others. MicroProfile Fault Tolerance provides features like bulkheads to achieve resource isolation and ensure critical components remain functional, even when others fail. + +==== Why Resource Isolation Matters + +In a distributed system, shared resources like thread pools, database connections, and network bandwidth can quickly become bottlenecks if not adequately managed. Resource isolation ensures: +- Failures in one service do not deplete resources for other services. +- Critical operations remain functional even under load or failure conditions. +- Better predictability and control over system behavior. + +==== Using Bulkheads to Isolate Resources + +Bulkheads are a common pattern for isolating resources by dividing a system into separate pools or partitions. This ensures that a failure in one area does not impact others. The MicroProfile Fault Tolerance standard provides the `@Bulkhead` annotation to implement this pattern. + +==== Bulkhead Types + +MicroProfile supports two types of bulkheads: +- **Semaphore-Style Bulkhead:** Limits the number of concurrent requests. +- **Thread Pool-Style Bulkhead:** Uses a dedicated thread pool to isolate operations. + +===== Semaphore-Style Bulkhead + +The semaphore-style bulkhead pattern limits the number of concurrent requests that can be processed by a service or method at any given time. Any additional requests are immediately rejected when the specified concurrency limit is reached. This approach prevents resource contention and protects the system from being overwhelmed during high traffic or failure scenarios. + +[source,java] +---- +package io.microprofile.tutorial.store.payment.service; + +import org.eclipse.microprofile.faulttolerance.Bulkhead; +import jakarta.enterprise.context.ApplicationScoped; + +@ApplicationScoped +public class PaymentService { + + /** + * Processes payment transactions with limited concurrency to prevent + * system overload and ensure stability during high traffic. + * + * The @Bulkhead annotation ensures that only a limited number of + * concurrent requests (5 in this case) can access this method. + * + * @return A success message indicating the processing status. + */ + @Bulkhead(value = 5) + public String processPayment() { + simulateDelay(); + return "Payment processed with limited concurrency."; + } + + private void simulateDelay() { + try { + Thread.sleep(1000); // Simulating a delay + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Error during payment processing simulation", e); + } + } +} +---- + +In this example: +- The method allows up to 5 concurrent invocations (`value = 5`). +- Any additional requests are rejected to prevent overload, ensuring system stability. + +===== Thread Pool-Style Bulkhead + +The thread-pool-style bulkhead pattern leverages a dedicated thread pool to achieve resource isolation. Incoming requests are placed into a queue when all threads in the pool are in use and executed as threads become available. This design helps manage resource contention effectively. + +[source,java] +---- +package io.microprofile.tutorial.store.payment.service; + +import org.eclipse.microprofile.faulttolerance.Bulkhead; +import jakarta.enterprise.context.ApplicationScoped; + +@ApplicationScoped +public class PaymentService { + + /** + * Processes payment transactions using an isolated thread pool to ensure + * stability during high traffic and prevent contention with other operations. + * + * The @Bulkhead annotation configures a thread pool with a maximum of + * 5 concurrent threads and a waiting task queue of size 10. + * + * @return A success message indicating the payment status. + */ + @Bulkhead(value = 5, waitingTaskQueue = 10) + public String processPayment() { + simulatePaymentProcessing(); + return "Payment processed successfully with an isolated thread pool."; + } + + private void simulatePaymentProcessing() { + try { + Thread.sleep(3000); // Simulating a resource-intensive task + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException("Error during payment processing simulation", e); + } + } +} +---- + +In this example: +- The method uses a thread pool with up to 5 concurrent threads (`value = 5`) and a queue of up to 10 tasks (`waitingTaskQueue = 10`). +- This configuration prevents failures in one operation from depleting shared resources. + +==== Externalizing Bulkhead Configuration + +Bulkhead resource limits can be externalized using MicroProfile Config to allow runtime adjustments. For example: + +Annotate the method without specific values: + +[source,java] +---- +@Bulkhead +public String processPayment() { + simulatePaymentProcessing(); + return "Payment processed successfully with an isolated thread pool."; +} +---- + +Define bulkhead parameters in `microprofile-config.properties`: + +[source,properties] +---- +com.example.Service/dynamicBulkheadOperation/Bulkhead/value=5 +com.example.Service/dynamicBulkheadOperation/Bulkhead/waitingTaskQueue=10 +---- + +==== Best Practices for Resource Isolation + +- **Isolate Critical Resources:** Use bulkheads for high-priority operations, such as authentication, to ensure they are not impacted by failures elsewhere. +- **Monitor Usage:** Track bulkhead metrics using MicroProfile Metrics to identify bottlenecks and adjust limits. +- **Plan for Scaling:** Test bulkhead configurations under various load conditions to ensure scalability. +- **Combine with Graceful Degradation:** Pair bulkheads with fallbacks to handle rejected requests gracefully. + +By effectively isolating resources, you can ensure that your microservices remain reliable and resilient, even in the face of unexpected failures or high demand. This approach not only protects critical operations but also improves overall system stability. + == Summary This chapter explored the MicroProfile Fault Tolerance API and essential fault tolerance strategies: From 08046de6b24ae99544a02e3dc8252c6adfff866e Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sun, 12 Jan 2025 19:48:47 +0530 Subject: [PATCH 07/12] Update chapter08.adoc --- chatper08/chapter08.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/chatper08/chapter08.adoc b/chatper08/chapter08.adoc index 4bbb7aaa..34f15514 100644 --- a/chatper08/chapter08.adoc +++ b/chatper08/chapter08.adoc @@ -560,6 +560,7 @@ By effectively isolating resources, you can ensure that your microservices remai == Summary This chapter explored the MicroProfile Fault Tolerance API and essential fault tolerance strategies: + - **Retries:** Automatically reattempt failed operations for transient errors. - **Timeouts:** Define maximum execution times for operations to avoid resource blocking. - **Circuit Breakers:** Prevent repeated calls to failing services and allow graceful recovery. From b305062d7c0410d15482c4fd6a8b2582ada9713e Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Mon, 13 Jan 2025 10:37:46 +0530 Subject: [PATCH 08/12] Create index.adoc --- chapter09/index.adoc | 481 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 481 insertions(+) create mode 100644 chapter09/index.adoc diff --git a/chapter09/index.adoc b/chapter09/index.adoc new file mode 100644 index 00000000..a9baa451 --- /dev/null +++ b/chapter09/index.adoc @@ -0,0 +1,481 @@ +== Chapter 9: MicroProfile OpenTelemetry + +This chapter focuses on distributed tracing and observability for Java-based microservices. In a microservices architecture, where requests traverse multiple services, understanding the flow of operations across these services is crucial for monitoring performance, diagnosing issues, and optimizing system behavior. MicroProfile OpenTelemetry enables developers to capture and visualize telemetry data, such as trace information and context propagation, ensuring transparency and enhancing the reliability of complex distributed systems. This chapter covers various topics about how to gain insights into the flow of requests and monitor the performance of service, and implement robust tracing mechanisms to optimize the behavior and reliability of microservices. + +=== Topics to be covered: + +* Introduction to MicroProfile OpenTelemetry +* Trancing Concepts (Spans, Traces and Context Propagation) +* Instrumenting OpenTelemetry +* Setting up Tracing Providers +* Context Propagation and Correlation +* Analyzing Traces +* Security Considerations for Tracing + +=== Introduction to MicroProfile OpenTelemetry + +MicroProfile OpenTelemetry is a specification that provides a vendor-neutral API for instrumenting, collecting, and exporting telemetry data from microservices applications. Built on the OpenTelemetry project, an open-source observability framework, it offers a comprehensive set of tools and libraries for collecting and analyzing telemetry data. By integrating seamlessly with MicroProfile, it allows developers to implement distributed tracing, metrics, and logging with minimal effort. + +MicroProfile OpenTelemetry makes it easy for developers to add tracing, metrics, and logging to their microservices applications. It provides a consistent API across different vendors, so developers can easily switch between different implementations without having to rewrite their code, promoting portability and reducing vendor lock-in. The specification supports capturing detailed telemetry data, such as trace information and context propagation, enabling better monitoring and debugging of distributed systems. This enhanced visibility is critical for optimizing system behavior, ensuring reliability, and improving the overall user experience in complex microservices architectures. + +=== Tracing Concepts (Spans, Traces and Context Propagation) + +Tracing is a powerful tool for understanding the flow of requests through a distributed system. Tracing concepts such as *spans*, *traces*, and *context propagation* are fundamental to achieving observability and gaining insights into the system's behavior. + +==== Spans + +A *span* is the basic unit of work in tracing. It represents a single operation or task performed by a service, such as an HTTP request, a database query, or a computation. Each span contains metadata, including: + +* *Operation Name*: Describes the activity (e.g., +HTTP GET /orders+). +* *Start Time and Duration*: Captures when the operation started and how long it took. +* *Attributes*: Key-value pairs providing context (e.g., user IDs, resource names, HTTP status codes). +* *Parent Span ID*: Indicates the parent span, forming a relationship within a trace. + +Spans may also include additional data like logs and events, which help provide a detailed view of the operation's lifecycle. Spans are connected together to form a trace, which provides a visual representation of the flow of a request. This can be helpful for identifying bottlenecks and performance issues. + +==== Traces + +A *trace* is a collection of related spans that represent the end-to-end execution of a request or transaction. It provides a holistic view of how a single request flows through the system, including interactions between services. Traces often form a tree structure, where the root span represents the entry point (e.g., a user request), and child spans represent subsequent operations. + +For example: + +* A user requests order details. +* The API gateway records the root span. +* Child spans are created as the request is forwarded to the order service, which interacts with a database. + +==== Context Propagation + +*Context propagation* ensures that trace metadata, such as trace IDs and span IDs, travels with requests as they move through different services and threads. This metadata allows spans created in different services to be correlated into a single trace. Context propagation is vital for connecting distributed spans and understanding their relationships. + + +In Java, context propagation is often implemented using thread-local variables or libraries like MicroProfile Context Propagation. This ensures that tracing information is maintained seamlessly across thread pools, asynchronous tasks, and service boundaries. + +Together, these concepts form the foundation of distributed tracing, enabling developers to monitor, analyze, and optimize the performance of their microservices effectively. + +=== Instrumenting OpenTelemetry + +Instrumenting Open Telemetry in an E-commerce Application + +*Step 1: Add the MicroProfile OpenTelemetry Dependency* + +First, include the necessary dependencies for MicroProfile OpenTelemetry in your `pom.xml` file. These dependencies enable tracing and exporting of telemetry data. + +[source, xml] +---- + + org.microprofile.opentelemetry + microprofile-opentelemetry-api + 1.8.0 + +---- + +*Step 2: Create a Tracer* + +A Tracer is used to create spans and manage trace data within the application. Initialize a Tracer instance for your application: + +[source, java] +---- +import io.opentelemetry.api.trace.Tracer; +import io.opentelemetry.api.GlobalOpenTelemetry; + +//… +//… + +Tracer tracer = TracerFactory.getInstance().getTracer("my-tracer"); +---- + +*Step 3: Create a Span* + +Use the Tracer to create a span that represents a specific operation or activity in your application: + +[source, java] +---- +Span span = tracer.spanBuilder("my-span").startSpan(); +---- + +*Step 4: Add Attributes to the Span* + +Attributes provide additional metadata about the span, such as the HTTP method or the resource being accessed. This helps in contextualizing the trace data: + +[source, java] +---- +span.setAttribute(++"http.method"++, ++"+*+GET+*+"++); +span.setAttribute(++"http.url"++, ++"/products/12345"++); +span.setAttribute(++"user.id"++, ++"98765"++);+ +---- + +*Step 5: End the Span* + +When the operation completes, end the span to capture the telemetry data: + +[source, java] +---- +span.end(); +---- + +*Step 6: Export the Traces* + +To export traces to a backend like Jaeger, include the exporter dependency and configure the properties: + + +Add the Jaeger Exporter Dependency: + +[source, xml] +---- + + org.microprofile.opentelemetry + microprofile-opentelemetry-exporter-jaeger + 1.8.0 + +---- + +*Step 7: Configuration * + +Configure the exporter in `application.properties`: + +[source, properties] +---- +otel.traces.exporter=jaeger +otel.exporter.jaeger.endpoint=http://localhost:14268/api/traces +---- + +*Step 8: Verify the Traces* + +After implementing tracing, verify that the traces are being collected and exported: + +. Start the Jaeger server (or your chosen backend). +. Open the Jaeger UI at http://localhost:16686[http://localhost:16686]. +. Search for traces associated with your application and confirm that the telemetry data is visible. + +=== Setting up Tracing Providers + +MicroProfile OpenTelemetry supports multiple tracing providers, for exporting and analyzing traces. The default tracing provider is Jaeger, but developers can also use other providers such as Zipkin or OpenCensus. + +To set up a tracing provider: + +. Add the appropriate dependency to their project's pom.xml file. For example, to use Jaeger, developers would add the following dependency: + +[source, xml] +---- + + org.microprofile.opentelemetry + microprofile-opentelemetry-exporter-jaeger + 1.8.0 + + +---- + +Once the dependency has been added, configure the tracing provider by setting the following properties in the project's application.properties file: + +[source, xml] +---- +otel.traces.exporter=jaeger +otel.exporter.jaeger.endpoint=http://localhost:14268/api/traces +---- + +The `otel.traces.exporter` property specifies the tracing provider to use, and the `otel.exporter.jaeger.endpoint` property specifies the endpoint of the Jaeger collector. + +*Switching to Other Providers* + +To switch to another tracing provider, replace the Jaeger dependency with the appropriate exporter dependency, such as *Zipkin*, and update the configuration properties accordingly. + +By following these steps, you can effectively instrument your application with OpenTelemetry, enabling comprehensive observability and traceability for your microservices. + +=== Context Propagation and Correlation + +*Context propagation* refers to the mechanism of carrying trace-related metadata, such as *trace IDs* and *span IDs*, across service and thread boundaries. This ensures that all spans created during a request can be linked together to form a complete trace. + +*How it works* + +. *Trace Context*: Metadata that includes the +traceId+, +spanId+, and sampling information. +. *Propagation Mechanisms*: Trace context is typically carried in HTTP headers (e.g., +traceparent+) or message properties in message queues. +. *MicroProfile Integration*: MicroProfile Context Propagation seamlessly integrates with OpenTelemetry to ensure that the trace context is maintained across service calls. + +*Example: Propagating Context Across HTTP Requests* + +When making an HTTP request, the trace context is propagated using headers: + +[source, java] +---- +import jakarta.ws.rs.client.Client; +import jakarta.ws.rs.client.ClientBuilder; + +Client client = ClientBuilder.newClient(); +client.target("http://inventory-service/api/check") + .request() + .header("traceparent", "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01") + .get(); +---- + +In this example, the +traceparent+ header ensures that the trace context is passed to the downstream +inventory-service+. + +==== Correlation + +*Correlation* is the process of associating related spans and traces across multiple services and threads to form a cohesive view of a transaction. Correlation enables developers to: + +* Identify the source of bottlenecks or errors in distributed systems. +* Understand the dependencies and interactions between services. + +==== Trace and Span IDs + +* *Trace ID*: A unique identifier shared across all spans in a single trace. +* *Span ID*: A unique identifier for a single span. It is linked to a parent span, forming a hierarchy. + +*Example: Correlating Logs with Traces* + +By including trace and span IDs in logs, you can correlate logs with traces to gain deeper insights: + +[source, java] +---- +import org.slf4j.MDC; + +MDC.put("traceId", "4bf92f3577b34da6a3ce929d0e0e4736"); +MDC.put("spanId", "00f067aa0ba902b7"); + +log.info("Fetching product details for productId=12345"); +---- + +When viewing logs, the +traceId+ and +spanId+ allow you to link specific log entries to the corresponding spans in your tracing system. + +==== Context Propagation in Asynchronous Flows + +In asynchronous programming, maintaining context across threads is challenging. MicroProfile Context Propagation helps by enabling trace context to be passed seamlessly across asynchronous tasks. + +*Context Propagation in Async Tasks* + +[source, java] +---- +import org.eclipse.microprofile.context.ThreadContext; +import java.util.concurrent.CompletableFuture; + +ThreadContext threadContext = ThreadContext.builder().build(); + +CompletableFuture.runAsync(threadContext.contextualRunnable(() -> { + Span span = tracer.spanBuilder("async-task").startSpan(); + try { + // Perform async operations + } finally { + span.end(); + } +})); +---- + +This ensures that the trace context is preserved, allowing the spans created in the asynchronous task to be linked correctly to the trace. + +==== Best Practices for Context Propagation and Correlation + +. *Propagate Context Consistently: *Use standard headers like traceparent for HTTP and custom headers for other protocols. +. *Log Trace Identifiers: *Include trace and span IDs in logs to correlate logs and traces effectively. +. *Use Context Propagation Libraries:* Leverage tools like MicroProfile Context Propagation to simplify the management of context in asynchronous flows. +. *Secure Context Data: *Ensure that trace metadata does not include sensitive information and is transmitted securely. + +By leveraging context propagation and correlation, developers can gain a unified view of distributed transactions, enabling effective debugging and optimization of microservices. + +=== Analyzing Traces + +Once trace data is collected and exported to a backend system, analyzing these traces becomes a crucial step in understanding the behavior of your distributed microservices architecture. By examining traces, you can gain insights into system performance, identify bottlenecks, and detect failures or anomalies. + +==== Steps to Analyze Traces + +===== 1. Visualizing Traces + +Tracing backends like *Jaeger*, *Zipkin*, or *OpenTelemetry Collector* provide visual interfaces to explore and analyze traces. These tools display traces as timelines or dependency graphs, making it easier to: + +* Understand the sequence of operations. +* Identify the services and components involved in a request. +* Observe how requests propagate through the system. + +*Example in Jaeger:* + +* Open the Jaeger UI at +http://localhost:16686+. +* Search for traces using parameters like operation name, time range, or service. +* View a detailed breakdown of each span within the trace, including timing and attributes. + +===== 2. Identifying Bottlenecks + +Traces highlight spans with long durations or repeated retries, which often point to bottlenecks or inefficiencies. Pay close attention to: + +* *Critical Path*: The longest path in a trace that determines the total response time. +* *Service Dependencies*: Examine how upstream and downstream services interact to find slow components. +* *Retries and Failures*: Repeated spans or high failure rates indicate problematic dependencies or transient errors. + +===== 3. Diagnosing Failures + +Traces provide valuable information for diagnosing failures, including: + +* *Error Codes*: Look for spans with error attributes, such as `http.status_code=500`. +* *Exception Details*: Many tracing systems capture stack traces or error messages in spans. +* *Service Impact*: Identify which upstream and downstream services are affected by the failure. + +===== 4. Understanding Service Dependencies + +Dependency graphs generated from traces show the interactions between services. These graphs help: + +* Visualize which services depend on each other. +* Detects circular dependencies or excessive coupling. +* Plan optimizations by focusing on critical services. + +===== 5. Correlating Traces with Logs and Metrics + +Traces, when combined with logs and metrics, provide a comprehensive picture of the system: + +* *Logs*: Use trace IDs and span IDs in logs to correlate application logs with specific spans. +* *Metrics*: Correlate trace performance data with system metrics like CPU usage, memory consumption, or request rates. +Example: If a span indicates high latency, check corresponding logs and metrics to identify the underlying cause, such as a resource constraint or network delay. + +==== Tools for Trace Analysis + +===== Jaeger + +* Provides detailed timelines for traces. +* Offers a dependency graph visualization. +* Supports searching and filtering based on trace attributes. + +===== Zipkin + +* Focuses on simplicity and quick trace searches. +* Integrates with multiple programming languages and tracing libraries. + +===== OpenTelemetry Collector + +* Centralizes trace collection and routing to different backends. +* Supports advanced features like sampling and transformation. + +==== Best Practices for Analyzing Traces + +. *Establish Baselines*: Use traces to establish performance baselines for services. +. *Monitor Critical Paths*: Focus on traces that traverse critical services or user-facing operations. +. *Use Sampling Strategically*: Balance trace volume and storage costs by sampling traces intelligently. +. *Automate Alerts*: Set up alerts for abnormal patterns in traces, such as increased latency or failure rates. +. *Collaborate Across Teams*: Share trace insights with development, operations, and QA teams to improve system reliability. + +By analyzing traces effectively, you can identify opportunities to optimize your microservices, ensure smoother operations, and enhance the overall user experience. Tracing tools provide a powerful way to visualize and understand the intricate dynamics of distributed systems. + +When analyzing traces, developers should look for the following: + +* *Long spans:* Spans that take a long time to complete may indicate a performance issue. +* *Missing spans:* Missing spans can make it difficult to understand the flow of a request. +* *Errors:* Errors can indicate problems with a service or a request. +* *High latency:* High latency can indicate a problem with the network or a service. + +By analyzing traces, developers can identify and troubleshoot problems with their microservices applications. This can help developers improve the performance and reliability of their applications. + +Here are some tips for analyzing traces: + +* *Use a trace viewer:* A trace viewer is a tool that can help you visualize and analyze traces. +* *Look for patterns:* Look for patterns in the traces that may indicate a problem. +* *Correlate traces with metrics:* Correlate traces with metrics to get a better understanding of the performance of your application. +* *Use sampling:* Use sampling to reduce the number of traces that are collected. This can improve the performance of your tracing system. + +By following these tips, developers can effectively analyze traces to improve the performance and reliability of their microservices applications. + +==== Security Considerations for Tracing + +When implementing tracing in your applications, it is crucial to be mindful of security implications. Tracing involves collecting and storing data about application behavior, which can potentially expose sensitive information if not handled properly. + +* *Data Sensitivity:* Be cautious about the data included in traces. Avoid logging sensitive information such as passwords, API keys, or personally identifiable information (PII). +* *Access Control:* Implement strict access controls to limit who can view and manage trace data. +* *Encryption:* Consider encrypting trace data at rest and in transit to protect it from unauthorized access. +* *Storage:* Carefully manage the storage of trace data. Avoid storing traces indefinitely and implement data retention policies. +* *Third-Party Services:* If using third-party tracing services, ensure they have robust security measures in place to protect your data. + +===== 1. Avoid Capturing Sensitive Data + +Traces often include attributes and metadata that can contain sensitive information. Avoid storing or transmitting sensitive details, such as: + +* Personally Identifiable Information (PII) (e.g., names, addresses, social security numbers). +* Payment information (e.g., credit card numbers). +* Authentication credentials (e.g., passwords, API keys, tokens). + +*Best Practice:* + +Sanitize attributes before adding them to spans: + +[source, java] +---- +span.setAttribute("user.id", "anonymized-user-id"); +span.setAttribute("credit.card.last4", "****1234"); +---- + +===== 2. Encrypt Trace Data + +To prevent unauthorized access during transmission, ensure that telemetry data is encrypted. Use secure protocols such as HTTPS or TLS for exporting trace data to a backend. + + *Example:* + +* Configure the tracing provider to use encrypted connections: + +[source, properties] +---- +otel.exporter.jaeger.endpoint=https://secure-jaeger-collector.example.com +otel.exporter.otlp.endpoint=https://secure-collector.example.com +---- + +===== 3. Limit Trace Retention + +Trace data can grow rapidly in distributed systems. Retaining it indefinitely increases the risk of exposing sensitive information. Implement retention policies to: + +* Retain traces only for the necessary duration for debugging or performance analysis. +* Periodically purge older traces from storage. + +===== 4. Access Control and Auditing + +Restrict access to trace data to authorized personnel only. Ensure that your tracing backend implements robust authentication and authorization mechanisms. + +*Best Practice:* + +* Use role-based access control (RBAC) to define permissions for viewing and managing traces. +* Audit access to trace data regularly to identify potential misuse or breaches. + +===== 5. Sampling Strategies to Minimize Exposure + +Sampling reduces the volume of traces collected and limits the exposure of sensitive data by capturing only a subset of requests. Common strategies include: + +* Random Sampling: Captures a fixed percentage of traces. +* Rate-Limiting Sampling: Limits the number of traces per second. +* Key-Based Sampling: Samples traces based on specific attributes (e.g., user ID). + +*Example:* + +Configure sampling to capture traces for debugging specific operations: + +[source, properties] +---- +otel.traces.sampler=traceidratio +otel.traces.sampler.traceidratio=0.1 +---- + +===== 6. Compliance with Regulations + +Ensure that your tracing practices comply with data protection and privacy regulations such as GDPR, CCPA, or HIPAA. Key considerations include: + +* Anonymizing sensitive data before tracing. +* Informing users about telemetry collection in your privacy policy. +* Providing mechanisms to opt out of tracing where required. + +===== 7. Isolate Tracing Infrastructure + +The tracing infrastructure, such as Jaeger or OpenTelemetry Collector, should be isolated from the public internet and accessible only within secure networks. + +*Best Practice:* + +* Deploy tracing backends in private subnets or behind firewalls. +* Use VPNs or dedicated connections for remote access to tracing dashboards. + +===== 8. Monitor and Alert on Trace Anomalies + +Tracing can help detect potential security incidents. Monitor traces for unusual patterns, such as: + +* Unexpected spikes in requests. +* Requests from unknown or unauthorized sources. +* Abnormal response times indicating possible exploits. +Set up alerts for these anomalies to investigate and mitigate potential issues. + +By following these security considerations, you can leverage the benefits of distributed tracing without compromising the security of your system or the privacy of your users. Careful handling of trace data, coupled with robust encryption, access controls, and compliance practices, ensures that tracing remains a valuable yet secure component of your observability strategy. + +=== Conclusion + +MicroProfile OpenTelemetry provides a robust foundation for observability in Java-based microservices, enabling developers to implement distributed tracing seamlessly. By leveraging this specification, you can gain deep insights into the flow of requests, identify bottlenecks, and enhance the reliability and performance of your applications. The integration of standardized tracing concepts like spans, traces, and context propagation ensures that developers can maintain a cohesive understanding of their system's behavior across service boundaries. + +Through instrumentation, context propagation, and effective trace analysis, MicroProfile OpenTelemetry simplifies the complexities of monitoring and debugging distributed systems. It empowers teams to proactively address issues, optimize performance, and improve the user experience. Moreover, by adhering to security best practices, developers can ensure that telemetry data is protected, compliant with regulations, and free of sensitive information. + +In this chapter, we explored the critical security considerations surrounding tracing within the MicroProfile OpenTelemetry framework. We emphasized the importance of safeguarding sensitive data by avoiding the inclusion of Personally Identifiable Information (PII) in trace spans. Additionally, we discussed the potential security risks associated with tracing in production environments and the significance of carefully managing sampling rates and data retention policies. By adhering to these security best practices, developers can harness the power of tracing for observability while ensuring the confidentiality and integrity of their applications. + +As microservices architectures continue to evolve, the ability to observe and trace system interactions will remain a critical factor in maintaining resilient and efficient applications. MicroProfile OpenTelemetry stands as a valuable tool in achieving these goals, providing developers with the observability they need to deliver reliable, high-performance microservices in modern cloud-native environments. + + From 6b6c8a4551dccec733f7ea197cae4aaad87e6a68 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sat, 25 Jan 2025 05:19:35 +0530 Subject: [PATCH 09/12] Update index.adoc --- chapter09/index.adoc | 74 +++++++++++++------------------------------- 1 file changed, 22 insertions(+), 52 deletions(-) diff --git a/chapter09/index.adoc b/chapter09/index.adoc index a9baa451..d4bcd45b 100644 --- a/chapter09/index.adoc +++ b/chapter09/index.adoc @@ -1,22 +1,22 @@ -== Chapter 9: MicroProfile OpenTelemetry +== Chapter 9: MicroProfile Telemetry -This chapter focuses on distributed tracing and observability for Java-based microservices. In a microservices architecture, where requests traverse multiple services, understanding the flow of operations across these services is crucial for monitoring performance, diagnosing issues, and optimizing system behavior. MicroProfile OpenTelemetry enables developers to capture and visualize telemetry data, such as trace information and context propagation, ensuring transparency and enhancing the reliability of complex distributed systems. This chapter covers various topics about how to gain insights into the flow of requests and monitor the performance of service, and implement robust tracing mechanisms to optimize the behavior and reliability of microservices. +This chapter focuses on distributed tracing and observability for Java-based microservices. In a microservices architecture, where requests traverse multiple services, understanding the flow of operations across these services is crucial for monitoring performance, diagnosing issues, and optimizing system behavior. MicroProfile Telemetry enables developers to capture and visualize telemetry data, such as trace information and context propagation, ensuring transparency and enhancing the reliability of complex distributed systems. This chapter covers various topics about how to gain insights into the flow of requests and monitor the performance of service, and implement robust tracing mechanisms to optimize the behavior and reliability of microservices. === Topics to be covered: -* Introduction to MicroProfile OpenTelemetry +* Introduction to MicroProfile Telemetry * Trancing Concepts (Spans, Traces and Context Propagation) -* Instrumenting OpenTelemetry -* Setting up Tracing Providers +* Instrumenting MicroProfile Telemetry +* Switching Tracing Providers * Context Propagation and Correlation * Analyzing Traces * Security Considerations for Tracing -=== Introduction to MicroProfile OpenTelemetry +=== Introduction to MicroProfile Telemetry -MicroProfile OpenTelemetry is a specification that provides a vendor-neutral API for instrumenting, collecting, and exporting telemetry data from microservices applications. Built on the OpenTelemetry project, an open-source observability framework, it offers a comprehensive set of tools and libraries for collecting and analyzing telemetry data. By integrating seamlessly with MicroProfile, it allows developers to implement distributed tracing, metrics, and logging with minimal effort. +MicroProfile Telemetry is a specification that provides a vendor-neutral API for instrumenting, collecting, and exporting telemetry data from microservices applications. Built on the OpenTelemetry project, an open-source observability framework, it offers a comprehensive set of tools and libraries for collecting and analyzing telemetry data. By integrating seamlessly with MicroProfile, it allows developers to implement distributed tracing, metrics, and logging with minimal effort. -MicroProfile OpenTelemetry makes it easy for developers to add tracing, metrics, and logging to their microservices applications. It provides a consistent API across different vendors, so developers can easily switch between different implementations without having to rewrite their code, promoting portability and reducing vendor lock-in. The specification supports capturing detailed telemetry data, such as trace information and context propagation, enabling better monitoring and debugging of distributed systems. This enhanced visibility is critical for optimizing system behavior, ensuring reliability, and improving the overall user experience in complex microservices architectures. +MicroProfile Telemetry makes it easy for developers to add tracing, metrics, and logging to their microservices applications. It provides a consistent API across different vendors, so developers can easily switch between different implementations without having to rewrite their code, promoting portability and reducing vendor lock-in. The specification supports capturing detailed telemetry data, such as trace information and context propagation, enabling better monitoring and debugging of distributed systems. This enhanced visibility is critical for optimizing system behavior, ensuring reliability, and improving the overall user experience in complex microservices architectures. === Tracing Concepts (Spans, Traces and Context Propagation) @@ -50,20 +50,20 @@ In Java, context propagation is often implemented using thread-local variables o Together, these concepts form the foundation of distributed tracing, enabling developers to monitor, analyze, and optimize the performance of their microservices effectively. -=== Instrumenting OpenTelemetry +=== Instrumenting MicroProfile Telemetry Instrumenting Open Telemetry in an E-commerce Application -*Step 1: Add the MicroProfile OpenTelemetry Dependency* +*Step 1: Add the MicroProfile Telemetry Dependency* -First, include the necessary dependencies for MicroProfile OpenTelemetry in your `pom.xml` file. These dependencies enable tracing and exporting of telemetry data. +First, include the necessary dependencies for MicroProfile Telemetry in your `pom.xml` file. These dependencies enable tracing and exporting of telemetry data. [source, xml] ---- - org.microprofile.opentelemetry + org.eclipse.microprofile.telemetry microprofile-opentelemetry-api - 1.8.0 + 2.0.1 ---- @@ -120,9 +120,9 @@ Add the Jaeger Exporter Dependency: [source, xml] ---- - org.microprofile.opentelemetry - microprofile-opentelemetry-exporter-jaeger - 1.8.0 + io.opentelemetry + opentelemetry-exporter-jaeger + 1.34.1 ---- @@ -144,40 +144,12 @@ After implementing tracing, verify that the traces are being collected and expor . Open the Jaeger UI at http://localhost:16686[http://localhost:16686]. . Search for traces associated with your application and confirm that the telemetry data is visible. -=== Setting up Tracing Providers +=== Switching to Tracing Providers -MicroProfile OpenTelemetry supports multiple tracing providers, for exporting and analyzing traces. The default tracing provider is Jaeger, but developers can also use other providers such as Zipkin or OpenCensus. - -To set up a tracing provider: - -. Add the appropriate dependency to their project's pom.xml file. For example, to use Jaeger, developers would add the following dependency: - -[source, xml] ----- - - org.microprofile.opentelemetry - microprofile-opentelemetry-exporter-jaeger - 1.8.0 - - ----- - -Once the dependency has been added, configure the tracing provider by setting the following properties in the project's application.properties file: - -[source, xml] ----- -otel.traces.exporter=jaeger -otel.exporter.jaeger.endpoint=http://localhost:14268/api/traces ----- - -The `otel.traces.exporter` property specifies the tracing provider to use, and the `otel.exporter.jaeger.endpoint` property specifies the endpoint of the Jaeger collector. - -*Switching to Other Providers* +MicroProfile Telemetry supports multiple tracing providers, for exporting and analyzing traces. The default tracing provider is Jaeger, but developers can also use other providers such as Zipkin or OpenCensus. To switch to another tracing provider, replace the Jaeger dependency with the appropriate exporter dependency, such as *Zipkin*, and update the configuration properties accordingly. -By following these steps, you can effectively instrument your application with OpenTelemetry, enabling comprehensive observability and traceability for your microservices. - === Context Propagation and Correlation *Context propagation* refers to the mechanism of carrying trace-related metadata, such as *trace IDs* and *span IDs*, across service and thread boundaries. This ensures that all spans created during a request can be linked together to form a complete trace. @@ -470,12 +442,10 @@ By following these security considerations, you can leverage the benefits of dis === Conclusion -MicroProfile OpenTelemetry provides a robust foundation for observability in Java-based microservices, enabling developers to implement distributed tracing seamlessly. By leveraging this specification, you can gain deep insights into the flow of requests, identify bottlenecks, and enhance the reliability and performance of your applications. The integration of standardized tracing concepts like spans, traces, and context propagation ensures that developers can maintain a cohesive understanding of their system's behavior across service boundaries. - -Through instrumentation, context propagation, and effective trace analysis, MicroProfile OpenTelemetry simplifies the complexities of monitoring and debugging distributed systems. It empowers teams to proactively address issues, optimize performance, and improve the user experience. Moreover, by adhering to security best practices, developers can ensure that telemetry data is protected, compliant with regulations, and free of sensitive information. - -In this chapter, we explored the critical security considerations surrounding tracing within the MicroProfile OpenTelemetry framework. We emphasized the importance of safeguarding sensitive data by avoiding the inclusion of Personally Identifiable Information (PII) in trace spans. Additionally, we discussed the potential security risks associated with tracing in production environments and the significance of carefully managing sampling rates and data retention policies. By adhering to these security best practices, developers can harness the power of tracing for observability while ensuring the confidentiality and integrity of their applications. +MicroProfile Telemetry provides a robust foundation for observability in Java-based microservices, enabling developers to implement distributed tracing seamlessly. By leveraging this specification, you can gain deep insights into the flow of requests, identify bottlenecks, and enhance the reliability and performance of your applications. The integration of standardized tracing concepts like spans, traces, and context propagation ensures that developers can maintain a cohesive understanding of their system's behavior across service boundaries. -As microservices architectures continue to evolve, the ability to observe and trace system interactions will remain a critical factor in maintaining resilient and efficient applications. MicroProfile OpenTelemetry stands as a valuable tool in achieving these goals, providing developers with the observability they need to deliver reliable, high-performance microservices in modern cloud-native environments. +Through instrumentation, context propagation, and effective trace analysis, MicroProfile Telemetry simplifies the complexities of monitoring and debugging distributed systems. It empowers teams to proactively address issues, optimize performance, and improve the user experience. Moreover, by adhering to security best practices, developers can ensure that telemetry data is protected, compliant with regulations, and free of sensitive information. +In this chapter, we explored the critical security considerations surrounding tracing within the MicroProfile Telemetry framework. We emphasized the importance of safeguarding sensitive data by avoiding the inclusion of Personally Identifiable Information (PII) in trace spans. Additionally, we discussed the potential security risks associated with tracing in production environments and the significance of carefully managing sampling rates and data retention policies. By adhering to these security best practices, developers can harness the power of tracing for observability while ensuring the confidentiality and integrity of their applications. +As microservices architectures continue to evolve, the ability to observe and trace system interactions will remain a critical factor in maintaining resilient and efficient applications. MicroProfile Telemetry stands as a valuable tool in achieving these goals, providing developers with the observability they need to deliver reliable, high-performance microservices in modern cloud-native environments. From 1b8bf400384ec3656c45b77b7be0b8542f56e215 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sat, 25 Jan 2025 09:18:40 +0530 Subject: [PATCH 10/12] Rename chapter08.adoc to chapter08.adoc --- {chatper08 => chapter08}/chapter08.adoc | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {chatper08 => chapter08}/chapter08.adoc (100%) diff --git a/chatper08/chapter08.adoc b/chapter08/chapter08.adoc similarity index 100% rename from chatper08/chapter08.adoc rename to chapter08/chapter08.adoc From 31f97815382938703dffcccacff8c3fd8e622336 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sat, 25 Jan 2025 09:19:13 +0530 Subject: [PATCH 11/12] Create chapter10.adoc --- chapter10/chapter10.adoc | 1 + 1 file changed, 1 insertion(+) create mode 100644 chapter10/chapter10.adoc diff --git a/chapter10/chapter10.adoc b/chapter10/chapter10.adoc new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/chapter10/chapter10.adoc @@ -0,0 +1 @@ + From d12a20fe108beacf1c1b6b73a1d9de817e4a2bcd Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Sun, 26 Jan 2025 08:43:19 +0530 Subject: [PATCH 12/12] Update chapter10.adoc --- chapter10/chapter10.adoc | 112 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/chapter10/chapter10.adoc b/chapter10/chapter10.adoc index 8b137891..466666bc 100644 --- a/chapter10/chapter10.adoc +++ b/chapter10/chapter10.adoc @@ -1 +1,113 @@ +== Chapter 10: MicroProfile JWT Authentication + +In modern microservices architectures, securing communications between clients and services, as well as between individual services, is critical. JSON Web Token (JWT) authentication provides a lightweight and efficient mechanism to achieve this. MicroProfile JWT Authentication builds upon this standard, offering developers a seamless way to implement authentication and authorization in their Java microservices. + +This chapter focuses on JSON Web Token (JWT) Authentication in the context of MicroProfile, providing a secure and efficient method for authenticating and authorizing users in microservices architectures. + +Topics covered in this chapter include: +- JWT basics +- User authentication +- Claims +- Scopes +- Token expiration +- Role-based access control +- Endpoint security +- Integration with identity providers +- Token revocation +- Best practices + +=== Introduction to JWT Authentication + +This chapter explores the fundamentals of JWT authentication, its implementation in MicroProfile, and best practices for ensuring security and efficiency. Topics include defining and validating user claims, managing token lifecycles, integrating with identity providers, and handling advanced scenarios like token revocation and refresh. With these tools, developers can build secure, scalable, and maintainable authentication systems for their microservices. + +=== What is a JSON Web Token (JWT)? + +A JSON Web Token or JWT is an open standard that defines a compact and self-contained way of securely transmitting information between parties as a JSON object. JWTs can represent any claim, such as identity, security roles, or session information. These claims can be verified and trusted because they are digitally signed. JWTs can be signed using a secret key or a public/private key pair. They can also be used for authorization by allowing users to access specific resources only if they have a valid JWT. + +The MicroProfile JSON Web Token specification defines how JWTs can be used in a microservice environment, including how to verify and validate tokens. It builds on the JWT standard and adds some additional features that are specific to microservices. + +=== Security Best Practices for Microservices + +Microservices are all the rage these days. But with more services comes more complexity – and with more complexity comes a greater risk of security breaches. So how do you go about securing your microservices? + +There are a few basic steps you can take to get started. First, make sure that each service is properly authenticated and authorized. Second, use encryption to protect sensitive data as it moves between services. Third, employ robust monitoring and logging tools to detect and track suspicious activity. And finally, be sure to keep your software up-to-date with the latest security patches. + +Following these steps will help you secure your microservices against the most common attacks, but it’s important to stay vigilant and always watch for new threats. + +Let’s now look into controlling user and role access to microservices with MicroProfile JSON Web Token. + +=== Use cases for JSON Web Tokens + +JSON Web Tokens can be used in the following scenarios: + +==== Authentication + +JWTs can be used to authenticate users. A user can be authenticated by providing a JWT with the correct signature. In-memory authentication is faster and more convenient as user credentials are stored in memory, but it is less secure because the data is more vulnerable to attack. DB stored user credentials are slower and more challenging to use, but they are more secure because the data is better protected either by encrypting it or through access control. Encryption makes it more difficult for hackers to access and steal information. Access control ensures that only authorized users have permission to view or modify the data. +==== Authorization + +One common use case for JSON Web Tokens is an authorization. In this scenario, the JWT contains claims representing the user’s identity and security roles. These claims can determine what resources the user is allowed to access. +For example, a user with the admin security role might be allowed to access all resources. In contrast, a user with the read-only security role might only be allowed to access resources marked as readable. + +==== Session information + +Another common use case for JSON Web Tokens is storing session information. In this scenario, the JWT contains claims representing the user’s session information, such as when the session started and when it will expire. This information can track a user’s session and ensure that it does not exceed a certain time limit. + +==== Claims-based identity + +JWTs can also be used to represent claims-based identity. In this scenario, the JWT contains claims representing the user’s identity, such as their name and email address. These claims can be used by applications to identify the user. + +For example, an application might use the user’s email address claim to look up their profile information in a database. Or, an application might use the user’s name claim to display the user’s name on a welcome page. +JWTs can be used with other identity providers, such as LDAP or Active Directory, to provide a single sign-on experience for users. + +==== Information Exchange + +JWTs can also be used to exchange information between parties. In this scenario, the JWT contains claims representing the data being exchanged. For example, a JWT might contain an order id claim and a user id claim. These claims can be used by the application to look up the order and display it to the user. + +JWTs can exchange information and are often used in Single Sign-On (SSO) systems. + +==== Federation + +JWTs can also be used in federation scenarios. In this scenario, the JWT contains claims representing the user’s identity. These claims can be used by the application to identify the user and look up their profile information. + +JWTs can be used with other identity providers, such as Active Directory Federation Services (ADFS), to provide a single sign-on experience for users. + +=== Creating and Signing a JWT +A JWT consists of three parts: a header, a payload, and a signature. Each part separated from the others by a dot (`.`) like: + +[source] +---- +sdfdfsdfsdSDFJJEOKJA.0dsfDNFSUJSDJ.ASDSAasdhjasS +---- + +The header and payload are JSON objects, and the signature is a digital signature that can be used to verify that the JWT has not been tampered with. +The header typically contains information about the algorithm used to sign the JWT, and the payload contains the actual claims. The claims can be anything you want to represent, such as the user’s name, email address, or security role. + +The header of a JWT contains the following information: + +- The type of token (JWT) +- The algorithm used to create the signature (HS256) + +Here is an example header and payload: + +Header: +[source, json] +---- +{ +"typ": "JWT", +"alg": "HS256" +} +---- + +Payload: + +[source, json] +---- +{ +"iss": "Bob", +"exp": 1463808400, +"http://example.com/is_root": true, +"sub": "alice" +} +---- +