Testing the PD-1 Antibody, MK3475, Given With Ziv-aflibercept in Patients With Advanced Cancer

Inclusion Criteria: * In dose escalation, patients must have histologically or cytologically confirmed metastatic disease from any solid tumor (NOS 10029000). In dose expansion part 1, patients must have histologically or cytologically confirmed metastatic melanoma (10053571), renal cell carcinoma (NOS 10038485), ovarian cancer (NOS 10033272), or colorectal cancer (10009951,10038045). In dose expansion part 2, patients must have PD-1 resistant melanoma (10053571), PD-1 resistant renal cancer (NOS 10038485), or sarcoma (10061271) * Renal cell patients must have had at least one prior vascular endothelial growth factor (VEGF) tyrosine kinase inhibitor (TKI) * Ovarian cancer patients must be resistant to platinum therapy; therapy (i.e. within 6 months of last platinum therapy); patients who received greater than two prior platinum containing regimens will not be eligible * Patients with colorectal cancer should have failed at least one oxaliplatin-containing regimen * No more than two prior therapies for metastatic disease * Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of MK-3475 in combination with ziv-aflibercept in patients \< 18 years of age, children are excluded from this study, but will be eligible for future pediatric trials * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%) * Estimated life expectancy of greater than 6 months * Leukocytes \>= 2,000/mcL (within 10 days of treatment initiation) * Absolute neutrophil count \>= 1,500/mcL (within 10 days of treatment initiation) * Platelets \>= 100,000/mcL (within 10 days of treatment initiation) * Hemoglobin \>= 9 g/dL OR \>= 5.6 mmol/L (within 10 days of treatment initiation) * Serum total bilirubin =\< 1.5 X upper limit of normal (ULN) OR direct bilirubin =\< ULN for patients with total bilirubin levels \> 1.5 ULN (within 10 days of treatment initiation) * Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 X institutional ULN OR =\< 5 X ULN for patients with liver metastases (within 10 days of treatment initiation) * Serum creatinine =\< 1.5 X ULN or measured or calculated creatinine clearance (CrCl) \>= 60 mL/min for subject with creatinine levels \> 1.5 X institutional ULN (glomerular filtration rate \[GFR\] can also be used in place of creatinine or CrCl); creatinine clearance should be calculated per institutional standard (within 10 days of treatment initiation) * International normalized ratio (INR) or prothrombin time (PT) =\< 1.5 X ULN unless subject is receiving anticoagulant therapy not requiring laboratory monitoring as long as PT or partial thromboplastin time (PTT) is within therapeutic range of intended use of anticoagulants; therapeutic Coumadin is not acceptable (within 10 days of treatment initiation) * Activated partial thromboplastin time (aPTT) =\< 1.5 X ULN unless subject is receiving anticoagulant therapy not requiring laboratory monitoring as long as PT or PTT is within therapeutic range of intended use of anticoagulants (within 10 days of treatment initiation) * Urine protein-creatinine ratio (UPCR) =\< 1 on spot urinalysis or protein =\< 500 mg/24 hour urine * Archival tissue must be available or newly obtained core or excisional biopsy of a tumor lesion * Patients must have measurable disease based on Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 * The effects of MK-3475 and ziv-aflibercept on the developing human fetus are unknown; for this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; patients should continue contraceptive measures for 6 months from the last dose of all study medications * Female patients of childbearing potential should have a negative urine or serum pregnancy test within 24 hours prior to receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Female patients of childbearing potential should be willing to use 2 methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 120 days after the last dose of study medication; patients of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year * Should a woman become pregnant or suspect she is pregnant while she is participating in this study, she should inform her treating physician immediately; men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 4 months after completion of MK 3475 and ziv-aflibercept administrations * Ability to understand and the willingness to sign a written informed consent document * Dose expansion part 2 PD-1/PD-L1 resistant or refractory melanoma and renal cell cancer only: subjects must have received prior immunotherapy with an anti-PD-1 or anti-PD-L1 containing regimen and must have progressive or recurrent disease after prior PD-1/PD-L1 directed therapy; primary resistance is determined at the time of initial restaging from initiation of treatment, as evidenced by progression by RECIST 1.1; acquired resistance would be a subject who had a best overall RECIST response of stable disease, partial response, or complete response confirmed radiographically by a second scan who subsequently developed progressive disease by RECIST 1.1 at any time thereafter * Dose expansion part 2 sarcoma only: subjects must have received standard of care treatment Exclusion Criteria: * Patients who have had chemotherapy, targeted small molecule therapy, or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study or those who have not recovered from adverse events due to agents administered more than 4 weeks earlier * Note: patients with =\< grade 2 neuropathy are an exception to this criterion and may qualify for the study * Note: if patients received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy * Patients who are currently participating in or have participated in a study of an investigational agent or using an investigational device within 4 weeks of the first dose of treatment * Has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment * Has had a prior monoclonal antibody within 4 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events (AEs) due to agents administered more than 4 weeks earlier * Has a known additional malignancy that is progressing or requires active treatment; exceptions include basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or in situ cervical cancer that has undergone potentially curative therapy * Lesions suspected to be at higher-risk for bleeding such as bowel involvement with tumor that invades into the bowel wall or involves the intraluminal component of bowel by imaging or direct visualization or central pulmonary lesions * Ulcerated skin lesions * Full anti-coagulant therapy Coumadin; patients may be receiving therapeutic Lovenox, Fragmin, or other heparin product that does not require laboratory monitoring * Poorly-controlled hypertension as defined blood pressure (BP) \> 150/100 mmHg, or systolic (S) BP \> 180 mmHg when diastolic (D) BP \< 90 mmHg, on at least 2 repeated determinations on separate days within 3 months prior to study enrollment * Pregnant or nursing women * Patients with known brain metastases should be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events * Patients with carcinomatous meningitis should also be excluded * Patients with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least 3 months prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to trial treatment * History of allergic reactions attributed to compounds of similar chemical or biologic composition to MK-3475 and ziv-aflibercept * Has an active autoimmune disease requiring systemic treatment within the past 3 months or a documented history of clinically severe autoimmune disease, or a syndrome that requires systemic steroids or immunosuppressive agents; patients with vitiligo or resolved childhood asthma/atopy would be an exception to this rule; patients that require intermittent use of bronchodilators or local steroid injections would not be excluded from the study; patients with hypothyroidism stable on hormone replacement or Sjogren's syndrome will not be excluded from the study * Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial, interfere with the patient's participation for the full duration of the trial, or is not in the best interest of the patient to participate, in the opinion of the treating investigator * Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial * Dose escalation and dose expansion part 1 only: Has received prior therapy with an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CD137, or ziv-aflibercept (prior treatment with bevacizumab is not an exclusion criteria) * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, interstitial lung disease or active, non-infectious pneumonitis, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * If pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment; pregnant women are excluded from this study because MK-3475 is an agent with the potential for teratogenic or abortifacient effects; because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with MK-3475, breastfeeding should be discontinued if the mother is treated with MK-3475; these potential risks may also apply to other agents used in this study; MK-3475 may have adverse effects on a fetus in utero; furthermore, it is not known if MK-3475 has transient adverse effects on the composition of sperm; patients are excluded from this study if pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment * Men and non-pregnant, non-breast-feeding women may be enrolled if they are willing to use 2 methods of birth control or are considered highly unlikely to conceive; highly unlikely to conceive is defined as 1) surgically sterilized, or 2) postmenopausal (a woman who is \>= 45 years of age and has not had menses for greater than 2 years will be considered postmenopausal), or 3) not heterosexually active for the duration of the study; the two birth control methods can be barrier method or a barrier method plus a hormonal method to prevent pregnancy; patients should start using birth control from study visit 1 throughout the study period up to 120 days after the last dose of study therapy; the following are considered adequate barrier methods of contraception: diaphragm, condom (by the partner), copper intrauterine device, sponge, or spermicide; appropriate hormonal contraceptives will include any registered and marketed contraceptive agent that contains an estrogen and/or a progestational agent (including oral, subcutaneous, intrauterine, or intramuscular agents); patients should continue contraceptive measures for 6 months from the last dose of all study medications; patients should be informed that taking the study medication may involve unknown risks to the fetus (unborn baby) if pregnancy were to occur during the study; in order to participate in the study they must adhere to the contraception requirement (described above) for the duration of the study and during the follow-up period defined; if there is any question that a patient will not reliably comply with the requirements for contraception, that patient should not be entered into the study; pregnancy: If a patient inadvertently becomes pregnant while on treatment with MK-3475, the patient will immediately be removed from the study; the site will contact the patient at least monthly and document the patient's status until the pregnancy has been completed or terminated; the outcome of the pregnancy will be reported without delay and within 24 hours if the outcome is a serious adverse experience (e.g., death, abortion, congenital anomaly, or other disabling or life-threatening complication to the mother or newborn); the study investigator will make every effort to obtain permission to follow the outcome of the pregnancy and report the condition of the fetus or newborn; if a male patient impregnates his female partner the study personnel at the site must be informed immediately and the pregnancy reported and followed; it is unknown whether MK-3475 is excreted in human milk; since many drugs are excreted in human milk, and because of the potential for serious adverse reactions in the nursing infant, patients who are breast-feeding are not eligible for enrollment * Patients who are human immunodeficiency virus (HIV) positive may participate IF they meet the following eligibility requirements: * They must be stable on their anti-retroviral regimen, and they must be healthy from an HIV perspective * They must have a cluster of differentiation (CD)4 count of greater than 250 cells/mcL * They must not be receiving prophylactic therapy for an opportunistic infection * Has known active hepatitis B (e.g., hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected) * Has received a live vaccine within 30 days prior to the first dose of trial treatment * History within 6 months prior to treatment of myocardial infarction, severe/unstable angina pectoris, coronary artery bypass graft (CABG), New York Heart Association (NYHA) class III or IV congestive heart failure (CHF), stroke or transient ischemic attack (TIA) * History within 3 months prior to treatment of grade 3-4 gastrointestinal (GI) bleeding/hemorrhage, treatment resistant peptic ulcer disease, erosive esophagitis or gastritis, infectious or inflammatory bowel disease, diverticulitis, pulmonary embolus, or other uncontrolled thromboembolic event * Patients who are less than 4 weeks post-operative (op) after major surgery